Audio Programmer Virtual Meetup - 9 Feb at 1830 UTC

all right we're about to go live here okay and we're live take it away tamer you're live all right well hello everyone and welcome to the monthly audio programmer meetup um yeah thank you so much for tuning in thank you especially for the hang on let me mute this okay sorry for that uh yeah thank you very much also for the people who are coming back every month to uh thank you so much for following this um my name is timo domna i'm your host together with joshua hodge who's also here uh hi josh how's it going great to be here yeah yeah it's it's really awesome to be here and it's actually the 12th time that we're doing this so we're approaching our one year anniversary really fast which is going to be next month and in march because we started all of this in march 2020 um yeah um and as you know this is every second tuesday of the uh month at 6 30 uk time um so yeah we're delighted to have you here tonight we have again today um two awesome speakers for you uh today we have not really themed evening but i would say like one common theme i know that a lot of people who are tuning in are programming with juice so today we actually have um both of our speakers are kind of programming like outside of this ecosystem so we're going to be looking at like different uh things that people are doing kind of outside of the jews universe which i think i hope is going to be very interesting and insightful for all of you i'm really looking forward to uh to that um and yeah um i guess josh do you have anything to add to that yeah thank you very much and uh as timer mentioned we have two great guests today we have uh mark heath who is from nice systems and we have chris santoro from soundtoys who's joining us today and uh we just want to take a moment as always to thank our sponsors who helped support the audio programmer meetup and make it possible so we have juice the juice framework we have focusrite and we have sonic so thank you very much to those three for helping to support us also we're always looking for other people to speak at the meetups and we'd love to hear your ideas and uh and if you have something nice that you'd like to present you don't have to be a 20-year veteran in order to present at the meet-up and you can submit a proposal at the audioprogrammer.comsubmit uh also we have a thriving audio programmer community on discord so we've just broke 5 000 members on our discord uh you can uh come join us it's audio programmers of all different levels so even if you're just curious about code and what it's about uh come hang out with us uh you can join us on the audioprogrammer.com forward slash community and with that i think that we're ready to take things away so who's first teamer um so yeah i think uh as you said it's time for our first speaker already let's just dive straight in our first speaker is um heath is a software architect and developer at nice systems and uh he's actually currently specializing in azure and net and he'll talk to us about whether.net and c sharp and all of the stuff from this kind of ecosystem is actually any good for audio development and i'm pretty sure he knows what he's talking about because he's actually made an open source library for doing exactly that stuff so um you know i'm mark i'm very very happy uh honored to have you here tonight and i'm really looking forward to uh listening to what you have to say hello and welcome great thank you very much i'm just going to share my screen first and check that you can see that and sorry before we do that um one more thing can i ask one more thing yeah um um so um for those of us who are watching us on youtube uh please ask your questions to mark which we are going to uh be answering just after the call uh sorry after the talk um but if you do so could you please be so kind and prefix your question with the word question in capital letters so that it is easier for us in the chat to kind of see where all the questions are and kind of collate them so we can like ask mark all of these questions at the end thank you very much that's all i wanted to add okay that's great so just double check before i begin that you can actually see my screen and hear my voice it's always a bit weird doing these presentations virtually um but it's all good thumbs up for many yeah we're good great okay let's get going so first of all i just want to say what privilege it is and to be able to speak on this um on this meet up um i've been doing bits of audio programming for many years and a lot of that time particularly when i first got started i really felt like i was floundering in the dark not knowing what i was doing and communities like this where people share their knowledge is such a brilliant thing to enable new people to get into audio programming and the other thing i was going to say at the beginning is that often when i'm doing talks on programming my audience is people who know a lot about net and nothing about audio programming but i'm guessing that the audience tonight is kind of probably going to be the other way around if anything so i won't assume that you know much about.net i'll try and make this accessible to you and hopefully you'll you'll find this interesting so um a little bit about who i am i'm working as a software architect um building digital evidence management systems um in the cloud and i've got details of my blog here i create training courses for programmers at a company called pluralsight um and the three of those courses i've created there are are to do with audio programming on the.net framework um but the most relevant thing um that i'm going to be talking a bit about tonight is an open source library that i created um i started it almost 20 years ago for the dot-net framework called n-audio and so tonight i kind of plan to share with you some of the things that i learned along the way some of the challenges i ran into and hopefully you'll find the the journey interesting and so this is going to be a little bit of a story to begin with about how i even got into trying to do audio programming um with the dotnet framework and so if we go right back to the last millennium um back in the late 90s i was coding primarily in two languages or two groups of languages i was doing some a lot of c and c plus plus uh most things i was coding were the windows platform and i was also doing using things like visual basic and python and the reason back then we often coded in multiple languages is because they each have their different strengths and weaknesses and so i'm sure you've heard of the difference between compiled languages and interpreted languages and that c and c plus they're compiled directly into machine code whereas languages like python and visual basic your code is interpreted at runtime and so very much the way that i would always think about these languages back then was that if you wanted fast code if you if you're doing something real time and obviously audio um is a is very often a real time application you want to be in comp in the compiled world if you want to do low-level stuff so you want to interact with the operating system and do multi-threading and talk directly to hardware devices you kind of want to be in this compiled world as well but of course the downside of it is complexity c c and c plus plus um i'm sure many of you know give you many ways to shoot yourself in the foot um offers you less protection than some of these other languages against um silly mistakes whereas often people would um look down a little bit maybe on the interpreted languages as they're too slow to do anything real time you wouldn't want to try and make games or real-time audio for example in languages like that but they do have some benefits the operating system is abstracted away um from you making it much simpler and you can do what was called rad rapid application development so although maybe your program was a bit slower to run it was going to be much faster and easier to develop and then kind of about i think i don't know what year java was introduced but it was about that time just i discovered it late 90s and java kind of offered this interesting in between model of the two paradigms java as i'm sure some of you will know compiles um not directly into machine code but into bytecode which is um kind of machine code for an imaginary virtual machine and then what happens is a virtual machine the java virtual machine or runtime environment runs that on the particular system that you're on and that means that your compiled java byte code is very portable it can run on windows it can run on linux it just can run anywhere that the java virtual machine is and what the virtual machine is doing is what's called just in time compiling so as your program is running it's compiling that byte code into actual native machine code for the platform you're on and it does that very efficiently and so you're getting much closer to the speed of compiled languages but bringing many of the benefits from the interpreted world it also comes with a what's called a garbage collector so it's sorting out your memory management for you and we'll be talking more about that because that brings some interesting challenges into the world of audio programming and like many of the interpreted languages it comes with a whole um classed library kind of standard ways of doing everything and that was always a frustration for me in the world of c and c plus that it seemed every time i was programming in those languages i had to learn a new way of doing threading and a new way of doing string manipulation because there were so many different libraries whereas languages like java gave you one standard way of dealing with these kind of things so i was really quite excited by java but i didn't really imagine that it could be possibly used for anything like audio because it's still in my mind it was in the slow category of languages but i stumbled across this project called java layer and what java layer was is um and it came with a little player as well so you could play mp3s in real time entirely in java and that was going to be too slow here was pure java um that was decoding mp hopefully hearing me okay because motion is not great yeah your bandwidth seems to be glow things i did when i um started doing net programming was like so up you can find i've got a fully c-sharp mp3 decoder that's been completely rewritten actually since it was originally ported from java layer um but uh it was good for people who were developing on the windows or windows platform and that's primarily where my most of my development career has been writing windows software and things like java and python weren't initially at least very um friendly to run on windows they were written more with kind of a linux platform in mind initially and net brought many of the same ideas as java it actually gave you a choice of languages you could use a visual basic style syntax you could use a brand new language called c-sharp which has kind of become my main language over the years you could even use c plus although that was a bit clunky and annoying to work with trying to write net programs in c plus i tried it and i didn't really get on with it and you could even write net programs in java although microsoft ended up getting um i think sued and that that got completely discontinued um so i was very excited about net and one of the first things i did when i tried out the beta of.net kind of in the early 2000s was search through the class library to find like what audio capabilities has net got and i was very disappointed to find absolutely none there was there was no audio capabilities at all in net so i thought to myself well why don't i write an audio library which will have basically building blocks to enable you to write audio applications in.net because i knew from my experiences with java that it ought to be possible to do that and so i wanted to create wrappers for all of the windows audio apis all of the ways that you could play audio and record audio and windows and interact with sound cards so kind of setting the volume level of your sound card or picking which inputs that kind of thing i also wanted to be able to read and write wav files and midi files i was doing a lot of kind of manipulation of midi files at the time so i was diving into the specs of midi and working out how that worked and i also wanted to be able to play mp3s which was kind of why imported that java layer to c-sharp because back then it was the days when um i don't know if any of you remember winamp we were all ripping our cds to mp3s back in in those days um and playing them with winamp and so i wanted you to be able to do that kind of thing in net as well um and this was a very kind of ambitious project i kind of imagined that after a while someone else would come along and blow me out the water with a much bigger and better library but it hasn't really happened there's not actually been many people making audio libraries in.net so this project has gone on for many years but it really was just a hobby project i was doing it for fun i was learning a lot along the way getting a lot of things wrong but about four years later i got a job at nice and nice is where i still am at there so obviously i liked it is a nice place to work and nice were just about to embark on a brand new um development project and what they were going to build um was an incident reconstruction platform for the public safety market and what that means is by public safety i mean things think primarily police forces and air traffic control but also town councils and so on um and what this um product was going to do and it's it's still going it's been a very successful worldwide product for nice is it allows them it allows police forces and other public safety bodies to reconstruct an incident maybe that's something bad has happened and they want to go to court and they want to replay everything that happened so they want to replay all the audio and all the video and fit it together in a timeline and so we had lots of recordings of 999 calls or 9-1-1 in the usa lots of recordings of police radio so all of the radio between the dispatcher and the police cars and between the police officers is all recorded um and that can all be brought in along with the video footage things like car dash cam videos and put into a timeline and played back together and as i said this was nice's first kind of big product they were building on.net and i was i realized this was kind of the ideal use case for my open source audio library so i volunteered let me be the one who makes the audio playback engine for this application and you know if you'd asked me back then what my dream job was i guess my dream job would have been to do be an or you know an audio programmer and to create a digital audio workstation or vsts or something like that because that's always been my hobby um home studio recording and this wasn't really that but i actually had to build a lot of the types of features that you would need to build if you were building a digital audio workstation so this application could play audio at multiple speeds you could go down to quarter speed and all the way up to 16 speed because sometimes they wanted to just skim through hours of recordings and it had some basic effects that we had to do so they needed automatic gain control to um level out the volume levels of the phone calls it had to cope with um live streaming so the audio that we were playing we were often still downloading from the loggers and decompressing from the native codecs while we were playing it and one of the big challenges i faced was audio and video synchronization it's actually a really tricky problem to synchronize audio with video synchronizing audio with itself is is not actually that hard because you mix it all together before you give it to the sound card um but with video you've got to try and make the audio you're playing line line up with the video and a lot of these video players when you particularly when you ask them to reposition so if somebody jumps around in time the video players are often really bad at jumping to the time that you ask them to jump to they might jump to somewhere completely different and it's very hard to kind of recover from that and get everything back in sync now the mixing engine um won't impress you very much in terms of the bit depth it was 32 bits those of you who've dealt with mixing audio in code will know that you don't want to mix audio in pcm 16-bit samples you want to get to 32-bit floating point so it was a 32-bit mixing engine but the sample rate i worked at most of the time was eight kilohertz because most um telephone and radio audio recordings are recorded at a sample rate of eight kilohertz and although that might sound ridiculously low to you um in the human speech it's perfectly intelligible without a lot of the high-end frequency information the nyquist frequency if some of you have heard of that at eight kilohertz is four kilohertz so you it only records any information up to four kilohertz but that's perfectly fine to understand what someone is saying um i did actually eventually have to go up to 16 kilohertz because some of the radio codecs that the police started using like g722 were 16 kilohertz but that's about as high um hi-fi as it got this application the other thing that i had to do with this application was um a bit of audio waveform drawing and i have also open sourced some of my audio waveform libraries and because when when audio goes to court sometimes they need to redact it to bleep out people's names and addresses that can't be heard in a court case and so we had to give them the ability to bleep things out um so he did that probably the most famous use of this application was i don't know if any of you saw this film um there was a plane in america that took off and it hit a flock of gulls and it destroyed i think one of the engines and the pilot made the decision uh pilot sully he made the decision that rather than trying to get back to the airport he was going to land the plane on the river um i'm sure some of you remember that news story and so after that happened they got all of the recordings of the air traffic control and they got the black box out of the cockpit and got the audio that had been recorded from the cockpit and they pieced it together in this dot net audio reconstruction application as part of the investigation to decide whether um he'd made the right call or not and um i don't want to spoil the video for you if you've not watched the movie but he's vindicated in the end and considered a hero for making the right decision so it's kind of cool to know that my mixing engine had been used for something like that now to rewind a little bit when i realized that my open source audio code could be potentially useful for building this application it gave me a bit of a problem because i'm sure those of you who have um full-time jobs as programmers will know that part of your employment contract is likely to say that all of the code that you write during your plate your paid employment doesn't belong to you it belongs to the company that you're um writing the code for and i realized that this was going to cause a bit of a conflict if i just brought all of my audio code that i'd been writing in my spare time and dumped it into this product and then employees at the company also worked on the product um and i know open sources has taken off a lot more since the early 2000s but a lot of companies were very suspicious of open source back then um but i realized i did need to get this official so i i got my n-audio library up on codeplex which is now no more it's on a github now and i gave it a license and the license i gave it i wanted it to be a commercially friendly license one of the things you find with open source audio code is that it's very often gpl or lgpl and those licenses have a stipulation in them that if you use gpl or lgpl code you are obliged to open source the code that you of your application that consumes that code and for most uh commercial companies who consider their source code to be their intellectual property that's an absolute no-no so i went with a commercially um friendly license that would allow them to use my code without having to open source their intellectual property but it did mean that um nrdo has never been able to bring in any gpl other open source code i've had to find other ways of solving problems so we might talk a bit more about that later you just keep an eye on on the time right so i'm going to talk in this next section a bit about what it's what it's like to write audio code in c sharp um hopefully this will be of interest to some of you at least um and one of the first things that i needed to do with n audio was call all of the windows audio apis from c-sharp and the way that you do that in dot net is using a capability called p invoke or platform invoke and what that looks like i've given a little example here of here's how i would call a windows api that's called wave in get dev caps and all wave in dev caps is is an api that asks windows to tell you what are the capabilities of this recording device and in c-sharp what you do is you decorate it with what's called an attribute excuse me dll import that says which dll is this method in so it's in the win mm dll and then all you have to do is you have to match up so usually what you're doing you're looking at a header file a c header file and you're matching up the types and signature of the method in the header file and finding the corresponding c sharp types and over time you kind of learn how to map this but it can be a real pain to do because in the world of c they love their hash divines and so you're often not seeing the actual underlying type you're seeing things like l param and w param and you have to kind of chase down all the hash defines to work out what those are but in worlds of windows you often see what's called a d word double-sized word which means it's a 32-bit integer so you know int in c sharp if you see a pointer then you should use int pointer and i was a little bit sloppy about this initially because when i created an audio it was in the time of 32-bit operating systems and 32-bit processors and you could get away with modeling pointers and integers because they were both four bytes long but as soon as 64-bit came along i had to fix every place where i'd used an int instead of an in-pointer because in in the 64-bit world d words are still 32-bits but in pointers pointers are are sit now 64-bits long so it's important you get that kind of thing right the other thing that you might notice in here is the out keyword on the second parameter um when you pass a pointer to a c method to a native method this method is actually going to write into that pointer and so the out keyword is telling.net that it should expect the method it calls to actually write over the memory of the pointer that it's provided and then you have to define essentially again looking at the header file you have to define your own structure in c sharp that maps exactly to the structure in c and again it's mostly a case of just matching the types up but um you often need to add attributes as well so two of the attributes you can see here are struct layout in which which is telling.net that the way that it lays this structure out in memory has to remain exactly in the order that i've written the code so it's not allowed to optimize it by moving things around because i'm going to pass this structure to unmanaged code that is expecting these fields to be in this exact order and the other thing that you need to give it clues on is how to marshal strings because again if you're a c programmer you'll know there's many ways that you can pass strings around you can just pass a pointer but it may be that in your structure you've got an array of characters and those characters might be eight eight bit characters or they might be wide characters if you if you're on a unicode operating system and so this martial as attribute that you can see is telling the dot-net framework how it needs to marshal that string and in this case it can cope um if you get your definitions right it will cope with working on both 32-bit and 64-bit processors and on um 8-bits character and unicode character operating systems although we don't really care about that more anymore in windows because windows is unicode everywhere now let's talk about the garbage collector next and because this is this is kind of a big deal when we're thinking about audio the garbage collector is a really great capability of net 99 of the time or 99.9 percent of the time really um you don't need to worry about it it does exactly the right thing what the garbage collector's job is is to free up memory that you're no longer using so unlike in c where you have to remember for every malloc i must remember to do a free um most of the um languages like c-sharp will do your your memory management for you but it does it by means of this algorithm called the garbage collector and the garbage collector is very fast it uses a very advanced algorithm called a generational mark and sweep um algorithm and um i think testament to how fast this works is the fact that we've seen platforms like unity 3d which is a games development platform based on c-sharp have huge popularity and it's being used right across the industry for making games and the same is actually true of java minecraft uses java which is again a garbage collected language and so clearly garbage collection isn't an absolute show stopper for the gaming industry but it does pose some problems for audio and the two big problems are first of all the way that the garbage collector works is every while your program is running you can be allocating memory and it will just keep allocating memory that's fine but if at some point the net framework decides that a your program is using a lot of memory and b the system you're running on is short of memory it can trigger a garbage collection and when the garbage collection is triggered your program pauses and that means every thread in your program stops the garbage collection happens and then your program carries on now this is normally very fast in the order of a few milliseconds it's um it's very highly optimized code however i'm sure many of you who do home studio recording with digital audio workstations will know that it's quite common for people who do audio recording to want to work at latencies of sub 10 milliseconds which means that every 10 milliseconds i have to give the sound card another buffer of audio to play and if i miss a single one i'll hear a drop out which will sound like a pop in the audio and so it's a real challenge to get very low latency when you've got a garbage collector and so generally speaking i try and avoid low latency wherever possible to give myself that buffer of time that if the garbage collector hits i can still provide the sound card with the next buffer of audio without missing missing it the second problem and this is a real headache is that the garbage collector is actually allowed to move things in memory so after it's freed memory it can also decide to kind of defragment your memory and move all of the used memory together and all of the free memory next to each other and that's absolutely fine for net program because net programs don't deal directly with memory addresses they deal with references to objects so it doesn't matter if it's moved in memory but if i've given a pointer to a byte array of audio to my sound card and then the garbage collector moves that buffer to another location in memory i'm going to get a nasty crash um and so what are the solutions to these problems well the first recommendation i have is learn to write allocation-free code this is a good.net performance tip whatever kind of code you're writing and what i mean by that is um particularly with audio um you're often filling buffer after buffer with audio and giving constantly giving the sound card new buffers full of audio and instead of every time allocating a new buffer of audio just allocate two up front fill one with audio give it to the sound card and while it's playing that one you fill the other one and then you kind of swap them over so you're not allocating any new memory while you're playing audio and that means the garbage collector's got nothing to do now unfortunately if your application has got a user interface then it's very hard to stop that from allocating memory as the user's kind of scrolling around and clicking around on the display memory will end up getting allocated but generally speaking that's a good way to reduce the impact of garbage collection and if you are going to give native code the address of a managed object and managed just means it's some memory that the net framework has allocated rather than the operating system directly you have to pin those so that they don't move around in memory you tell the garbage collector you're not allowed to move this object or probably a better solution and i kind of wish i'd done this more and with hindsight is just instead of getting.net to allocate the memory just go directly to the operating system and do a malloc just like you would do in c and then you've got native operating system memory that's not going to move around okay am i still with you because i just saw a message saying my internet is poor oh you're good you're old good okay brilliant um so um i probably won't say too much about this because i don't want to bore you with lots of low-level details but um here's a couple of ways that you can pin memory in um in c-sharp and one of the interesting things is c-sharp has got this keyword called unsafe and the unsafe keyword is designed to scare you away and make you not want to use it and it's very good at that most c-sharp developers don't want to touch this but if you use the unsafe keyword what you're allowed to do is you're allowed to get pointers to managed objects and once you've got pointers to managed objects you can treat them just like you can treat pointers in c you can cast for example in this example i'm casting a bite array into a short array or a float array which is really convenient you need to be able to do that when you're working with audio because sometimes you've got 16-bit samples sometimes you've got 32-bit samples and so you need to be able to easily cast between buffer types but it also doesn't doesn't give you any of the protection that dotnet would normally give you in dot net it's impossible in normal code to write off the end of an array it won't let you do it but uh in c as i'm sure you know with the pointers you can write anywhere you like and you'll just um get access memory access violations if you do so and so it's possible to make those same kind of errors if you write code like this um then this code this slide is just a few examples of how you would use native memory um in c sharp so if you want the operating system to just give you memory you do um an alec h global and some of your windows programmers will be familiar with that api call and then um dotnet has got these marshall structure to pointer and pointer to structure methods that can copy from managed objects into unmanaged memory and back out again so i do that in a number of the methods in an audio i allocate native memory i copy into it i call the methods and then afterwards i copy out myself and then of course you're in charge of memory management so you've got to remember to free the memory when you're done with it or you get a memory leak now one of the challenges um or big decision points that i kept running into with an audio is what happens when i want um to some dsp or codex so say i want mp3 decoding or obvious or say i want something like a fast fourier transform or a resampler that we will talk about in a bit should i find a dll a native dll that does that and just write p invoke wrappers for it or should i find the source code for it and convert port the source code into c sharp and both of those options have got some strengths um and in some cases i ended up doing both um the advantages of interrupt to native code is at least in theory ought to run faster it's not actually necessarily so true anymore it is possible if you port algorithms from c to c sharp for them to run almost as fast but at least in theory the c code has been more heavily optimized than what you might manage to port yourself and probably one of the biggest advantages is if i write an interrupt wrapper for a dll and then a few months later they produce version two of that dll with bug fixes and new features um i can just easily take that and be up to date whereas if i've ported all of the c code to c sharp um and they've then bug fixed it i've then got to go and read the get diffs to work out what's changed and change my port to match it and it means that your code can end up lagging behind also it helps you avoid porting bugs although the c and c sharp languages look like they're very similar there are a number of really nasty gotchas in there when you're porting algorithms and i've done quite a lot of porting from c to c sharp and things like particularly bits manipulation if you're doing bitwise operators or bit shifting they can work subtly differently from how the c code works and so it's very easy to get weird bugs introduced if you're porting from one language to another but the big downside is portability because dotnet runs in many places it doesn't just run on 32-bit and 64-bit windows but it also runs on linux and with things like um unity that was talking about you can run c-sharp in android devices nintendo switches you can even run it in the web browser and so if you're relying on interrupter native code you can only use that in places where the native code is available so in many cases i actually found i needed to support all scenarios to also port to managed code and one of the advantages of porting to manage code is that i can then make the api surface area more idiomatic for programmers in c-sharp so it can feel more like a c-sharp library than then you're accessing kind of a c library in c sharp um but also porting means it's completely portable you can run in sandboxed environments um there are a number of environments c-sharp can run where you're not allowed to call native code so silverlight which is now dead was a way of running c-sharp in the browser and there's a new thing called blazer which is very similar but there's also uwp which was microsoft's platform for building those windows store apps i don't know if any of you've used the windows store but those apps are sandboxed they're not necessarily allowed to just call any api and you can't just randomly call um native code unless you've got permissions to break out of the sandbox and so having a fully managed port of your algorithm is really useful and like i said one of the big surprises to me of an audio was that people started using an audio in things like unity 3d which is a game across platform games library and people were using my library which i thought very much of as a windows only library but they were using it to make games that were running on android and running on um i say all these xbox and nintendo and so on and that meant that they could only use the bits of n audio that were cross-platform they couldn't use um anything that called into windows apis or native code so one of the examples of this just checking me time again we're still doing okay um was resampling so if you end up if you like me try and build yourself an audio library at some point you're going to want to re-sample audio between um two sample rates because if you know anything about mixing audio if you've got audio at two different sample rates you can't just mix it directly together you have to re-sample to a common sample rate you usually re-sample up to the highest common sample rate and then mix the audio together and so initially i thought well could i do it myself could i write my own resampling algorithm you know i did a bit of maths how hard could it be what you tend to find with a lot of audio dsp as i'm sure many of the viewers of this channel will know that it's often non-trivial to write good audio dsp algorithms yourself for something even seemingly as simple as re-sampling you need to learn about low-pass filters and implementing a good low-pass filter is critical to getting a good re-sampler so i started off by just making wrappers for native resamplers and windows actually comes with a whole bunch of resamplers that use various weird and wonderful apis there's a audio compression manager resampler that's a directx media object and a media foundation transform and i made wrappers for all of those so that meant that if you were on any version of windows that had any of those you could just excuse me you could just use that resampler however it still wasn't universally available i was playing with silverlight at the time and i couldn't use those in silverlight so i looked into well could i port an existing algorithm and there are a few communities that share audio algorithms on the internet so i don't know if any of you have seen the music dsp site um i think it's part of kvr now there were a bunch of resampling algorithms out there but they were almost always uh gpl which mean i couldn't touch them i couldn't just port them into an audio because it wasn't compatible with um writing commercial software and so eventually i found um the cockos wdl um resampler which had a license it had its own custom license that looked like it might be compatible so i reached out to them the folks over at reaper and i said would you mind if i ported your resampler into an audio and they were fine with it so i i've ported that resampler into an audio and that means that now i've got a fully managed um resampler now i'll probably not dwell too long on this slide some of you might have seen diagrams like this before which is a way of testing how good a resampler is you have a sine wave that goes from a low frequency to a high frequency then you re-sample it and this re-sampling on the right has got a little bit of aliasing just before the cutoff point but it's generally fairly good if you just try and write your own resampling algorithm like i did once without any really knowledge of what you're doing you'll end up with something that looks like this where you've just got horrendous aliasing sounds like weird and psychedelic you can just about hear what the original signal was but that's just um i thought i'd show you that to give you an idea that um there is a big learning curve in getting into audio dsp there's a lot of things you need to learn and that's why i think channels like this um communities like this are so valuable because um people are sharing the knowledge so the final thing i'm going to do in this section before maybe we we throw it open to some q a um i think i'm still in my time limit uh let's talk about some of the windows audio apis that i um converted because windows has got a bunch of different audio apis the first one is called win mn and this has been around for ages this was part of windows back in 1991 it's the first audio api that microsoft created for windows and it's still part of windows 10 today and the methods start with things like waving waveout and it's actually a very simple api to use one of its conveniences is that it's got built-in resampling which i found very convenient before i made my own resampler because it meant that if i had one audio file at eight kilohertz and one at 16 kilohertz i could just play them both with two instances of wave out and not worry about resampling myself it has a choice of callback models so usually with a sound card you give it audio and then it tells you when it's finished playing it or it tells you if you're recording it tells you when it's finished recording and you need to be notified in some way and respond to that and wave out can can do that in a bunch of different ways it can call a function directly it can post a windows message to the windows message pump or it can use um events mutexes and i i foolishly chose the wrong the wrong one um initially with an audio and found myself in all kinds of deadlocks because one of the things that you run into with audio is it really matters what thread you are on when you're doing stuff um you generally speaking want to be doing all of your audio processing in a single thread and your your ui in a different thread but you can get into a real mess if you kind of mix together some code that should be doing ui stuff is running in the audio threads or you're you're doing some some audio in one thread and then some other audio in a completely different thread you can get all kinds of deadlocks and problems but eventually i worked that out and the wave out supporting n audio is very stable and hasn't really needed to be fixed it for years um but it's quite high latency and by high latency i mean don't try and play audio with buffer sizes of less than about 50 milliseconds because you will get a stuttering um sound if you do that so it's not you're not going to use this to make a digital audio workstation but actually for a lot of applications and 50 milliseconds is absolutely fine and if you've just got a media player you're just playing back stuff [Music] most people will not even notice a 50 millisecond latency so that's absolutely fine for the vast majority of audio playback scenarios anyway in 2007 microsoft brought out windows vista and they it was quite a major overhaul to the whole of windows they were redoing lots of stuff they're redoing security and they're redoing drivers and they also completely from the ground up rewrote the audio api um for the to the windows audio session api um and vista kind of had a very bad name because lots of stuff broke then and just because so much had changed in windows and it took them a while to kind of restabilize but i was very excited about wazapi because i thought you know this is it this is the low latency um super audio api that is just going to be mean that n audio can be even faster and even better and it was actually a real challenge wizappi is a very large api surface area it took me a very long time to write wrappers for all of it i've still not done all of it um and it's also com based if any of you know about com in windows it's a it's a kind of a programming model that is actually quite painful to work with um from net and very opinionated on what thread you call from if you create a com object you have to call it from the same thread that you created it on or jump through hoops to be able to do that but this api was appy had two modes it had a shared mode which is like i'm playing audio but other applications might be playing audio and exclusive mode which is like i want kind of low level access i want my my audio to go with the minimum possible steps directly to the sound card and like i said i hoped this would be super low latency um my opinions of wazapi is that you can sometimes get really low latency but it still can be a challenge to do so but this api was ieee floating point right from the ground up you you always provide audio as floating point samples to it and you have to operate at whatever the um sample rate the sound card is using and we and this was another thing that forced me to have a good resampler in n audio because it won't resample for you if you're in shared mode um and the sound card happens to be going at 48 kilohertz and you want to play an mp3 which is 44.1 kilohertz you have to do the resampling yourself before you give the audio to wazapi that can actually make it painful to work with um but one really cool thing is that it had loopback capture which means that you can record the system output of your sound cards which you was always hard to do before windows vista and so that's a really great capability you can't still in windows record the output just of a single application without um kind of using custom third-party drivers to kind of make fake sound cards but you can record the whole system output and it also had some other quirks that made it hard to work with compared to win mm that the amount of audio that you provided each buffer could be different it wasn't always necessarily a fixed amount which made it tricky to work with then in 2016 as part of the universal windows platform which was this new api microsoft were creating to build windows store applications they introduced the audio graph and i was very excited about this at first because it's a really nicely designed api it's based on the concept of nodes so you kind of connect things together uh oh not sure what happened there hopefully i'll come back in a second uh okay yes um let's hope so i mean worst cases internet is cut out yeah that was pretty sudden yep well do we have a means of reaching him somehow and letting him know because otherwise he's just gonna keep talking uh yeah i'll email him um all right now he dropped off the zoom core yeah this actually in one year of doing these it has never happened to us before has it and that's the first time yeah yeah this is the first yeah this is the first time and um but wow what a fascinating talk i never yeah i don't think we've ever had a talk like that i think the closest thing we've had was the linux audio api talk from fabian yeah i think the thing that we need now is to have um like a core audio one about like macos and ios and then we cover we have covered everything you know yeah but that would that be you giving that talk then i'm not sure i'm up to date with the latest stuff but um let's think about that yeah great to see you mark yeah we saw you um for disappearing oh that's all right you want to continue on fascinating talk here yeah yeah really really enjoying this i think you cut out when you were explaining that um what wasapi uh uh what was it yeah um audiograph the loopback capture you were saying that you can capture the uh like the whole system output i don't think you uh i think you didn't mention the variable buffer size you were already gone at that point okay i'm sorry i talked for another couple of minutes then without realizing i was gone um okay so what i basically had said was um that was happy um unlike win mm um when you provide audio to the sound card it's not always exactly the same buffer size you're providing each time and that was a bit irritating to work around so i had to kind of rejig some of my code to cope with that then i the next audio api audio graph that came along in 2016 was for the universal windows platform which is basically microsoft's way of enabling you to make windows store applications and i was very excited about this one because it was a really nicely designed audio api it was based around the concept of an audio graph so um you know if you've got a guitar you plug your guitar into a effects pedal and then your effects pedal into an amp you know you connect things together like that this api was designed that way and i think it's a really good way of designing an audio api to connect inputs to effects to submixes and so on and it's based on top of wazapi um and the other nice thing about it was is that it was already exposed as a net api you could call it directly in c sharp without me needing to do any uh wrapping of it however um and i did i did play with this quite a lot i made a training course on pluralsight all about audiograph it does have a few weaknesses and limitations and so i feel like it's got potential um for net programmers interested in audio but it's not quite ready yet it's missing a few um bits that i would like and then the final one i'm sure you've all heard of this if you if you do any uh audio on windows is asio i call it i don't know how you're supposed to pronounce it this was created by steinberg way back in 1997 and to this day i would say it's still the de facto standard for um applications that need low latency audio to interact with the sound cards in windows most soundcard i don't know why but most sound card drivers just seem to be much better with azio than they are even with zappy which is kind of microsoft's theoretical best api for you to use um and you can often get very low latency now one of the great things about an audio being open source was that i got every now and then i'd get really great contributions from the community and a guy called alexander motel um a brilliant guy who's he's made a lot of really cool um open source projects himself contributed a a set of p invoke wrappers for asio for n audio so you can in an audio you can interact through the azio interface with your sound card and get the lowest possible latency that you could get um with the.net framework obviously bearing in mind what i said earlier about garbage collection and i think it's got a really nice elegant um api actually it's based on a buffer swap that is works with both recording and playback at the same time unlike most of the windows apis that treat recording and playback devices is completely separate with azio you get called back to be told that the sound card has finished playing a buffer and finished recording a buffer and that means that it's much easier to know how to synchronize what you played with what you recorded this is a very fixed timeline so it's a nice api it does have some weaknesses particularly if you want to use multiple sound cards it's not really got a good solution for that um but it's still probably the thing you want to use if you want ultra low latency audio um i'm going to skip past these slides a little bit and just mention because i'd like to have some time for questions um i don't know if any questions are appearing on the chat yeah yeah there are quite a few but like also you don't have to rush i think you if you you can take a few more minutes i think if this like material that you think is okay so i'll just show you a couple of bits of of the design of the api of event audio then and an audio i i tried to have a stream based api so the idea was that you created components that were streams of audio and you connected them together and just uses a very simple design pattern called the decorator pattern if any of you know about software design patterns um all of the classes you create implement this iwave provider interface and all a wave provider interface has is a wave format so it tells you what format the audio you're receiving is in and then a way of reading audio out and the idiomatic way of reading from a streaming.net is that you pass you have a read method with a byte array buffer and you ask it to read into your byte array you tell it what offset to write to and that's because you don't have pointers or you do um you don't tend to use pointers so you have to tell it what offset in the byte array and how many bytes you want it to fill and then it tells you how many bytes it wrote now that's great but actually most of the time your audio might start as pcm or a codec like mula or a law but pretty soon you convert it to floating point and so it's much more easy if you want to write effects in.net you don't want to be given a byte array because you can't cast a byte array to a float array so i had another interface called a sample provider that was a convenience interface that gave you the audio already as floating point samples and much easier to write effects and so on and mixing code like that now the casting problem like i said was a real headache for me because if you've declared a byte array i don't know if you can see my mouse pointer on the screen here but if you declare a byte array and it's actually full of floating point samples you can't do this you'll get a compile error just trying to cast it to float array and so there's a whole bunch of different things that you could do to try and get from a bite array to a floating point array but another really cool hack and this is one of my favorite c sharp hacks um that works amazingly well it feels like it's a bit dirty but somebody um came up with this idea that you could use um a structure with um and this is a special con construct that c sharp provides for interop when you're dealing with um unions so in c code if any of you've done c programming you'll know that often structs are unions that it might be this or it might be that and they both potentially take up the same memory space in the structure and we essentially did that with a bite array so you say there's a bite array but exactly the same location in the structure there's a a um float array and what that means is you can write bites into the bite array and then read floats out of the photore and vice versa and so that was a super good um improvement to an audio because it meant i didn't have to copy all the time or use other slower constructs it meant i had fast um casting however dotnet has moved on since then and quite recently i say quite recently about a couple of years ago they introduced this thing called the span of tea and this is kind of what i i've wished for all along in.net is that with a span of tea it's kind of like a pointer a safe way of doing pointers a span of tea can be a span of bites or a span of floats and what it basically means is it can point to either managed memory so memory allocated by net or to native unmanaged memory so memory that you've got with a h um and a span has got a length so it knows how much memory there is so it's safe it can't write over the end of the pointer and it also can be sliced um which just basically means you can do points to arithmetic but in a safe way i can pass around a span that just points to half of another span um and most most importantly it supports casting so if i've got a span of bytes i don't need to care whether it's native or managed memory and i can just cast it directly into a span of floats and then i can do my dsp but then i can cast it back into a span of bytes because that's what my sound card is expecting a byte pointer just makes life a lot easier and i've actually prototyped converting the whole event audio to use span of t it's in a branch in the github repo but i haven't made it the public api and the reason for that is um everyone would have to rewrite their code to use span of t instead of buy to raise or float arrays and that's a big breaking change and i tend to avoid breaking changes but maybe i can find a way in the future of bringing the benefits in for the people who want it without forcing it on everybody else a couple of other innovations that are happening in the dotnet world you may have heard of.net core so when the net framework originally came out it was just for windows it was very much a windows thing and that other people made linux implementations so if you've heard of mono that was a linux implementation of net however.net core is cross-platform you can now run c-sharp applications on linux and it has been much more highly performance tuned so there are huge performance gains if you're using the classic.net framework moving to.net core things are going a lot faster it's it's um it's actually a very competitive platform in terms of its performance now it's still got a garbage collector so the things i said about garbage collector are still true um but it's high performance and it's been picking up other features that are relevant if you're doing audio programming in particular function pointers are something that i wanted for a long time because um to call com based apis there are certain um op codes for the virtual machine that were not possible to generate directly with the c compiler and it meant that a lot of people who did audio programming and net would kind of have to do rewriting of the compiled code to change the under underlying op codes of the intermediate language it's called um cil common intermediate language that c-sharp compiles into and so this will just make things like that um less painful to work with and there's another project that is um i believe microsoft are actually going to do a twitch stream quite soon to announce this but this pla you can find it here on on github there's actually a series of of github projects um where they're trying to make it so that they auto generate wrappers for net to be able to call any windows api so all of that manual work that i did to write wrappers for hundreds of windows apis this is kind of promising if it goes well this project to do that automatically and they're not just doing it for net i think they want to do it for languages like rust and go as well so if you're using any of these languages and you want to call any windows api rather than having to write all the signatures yourself and go through the pain of working out how to do it this project is hoping to simplify that again it's something that had it been around at the start of an audio would have been absolutely ideal for me um but it's kind of a bit late now but maybe i can use it in the future so final slide before i jump to some questions can i use dot net for audio absolutely yes i've shown you tonight um the n audio toolkit that i made i talked to you about making a managed mp3 decoder again it's on github called n-layer another project that i did that i haven't talked about because it doesn't work anymore skype has unfortunately changed its api but i made a real-time voice changer application for skype and that could turn your voice into a robot voice and not do a funny effects on your voice while you were talking by intercepting the audio from your microphone and modifying it in dot-net and that was actually a super popular program i had millions of downloads of that i've made midi file drum remappers a number of the people who sell midi drum grooves use my libraries to turn the midi drum grooves into the formats of all the different drum samplers that there are and i've also shown you that for commercial enterprise applications it's not music application but many companies are using net to build audio applications for telephony and even medical care um i know an audio is being used in a number of healthcare scenarios as well which is really interesting and the final thing i'm going to say is a little bit sad in a way but in the last five years my day job has moved from working with applications that run on the client of windows to applications that are web browser-based and that means that i don't actually need c-sharp code that can play audio anymore because all my audio playback is happening in the web browser and so what that's meant for in audio is that it's not really moving on very fast i'm not working on it nearly as much as i used to and so i'm on the lookout for new maintainers so if you're passionate about audio programming and you love the dotnet framework and you'd like to get involved either as a contributor or even a maintainer of the project then feel free to reach out to me it'd be great to hear from you so i've just got a few links there to myself and the projects and the audio training courses for net programming um but um i think now is probably a good time to do any questions uh yeah thank you so much mark for this talk really really fascinating stuff really interesting like take on on making music i'm sorry making audio software um and yeah i definitely a lot of new new information there i had no idea you know that you could really do this stuff with csharp and dotnet and um in this depth as well and like you know that's really quite fascinating so thank you so much for sharing all of this um we do have quite a few questions from the audience um let's maybe start at the top so um steve dwyer is asking which open source license model did you eventually use so um initially i used a license called mspl and microsoft as they as they want to do they like to make their own version of everything and they made their own open source licenses and the mspl was the microsoft permissive license however i realized that most people didn't know what that meant in the open source world because it's a bit of an unusual one so i've actually switched fairly recently to mit because that's very well known and it's more or less the same kind of philosophy it's commercially friendly license yeah mit i think is pretty pretty popular thanks so um we have another question from the monty music channel um did you have to ever wrap old c and c plus plus code for your library for low level access and i guess what they mean is not wrapping like cpus apis because obviously that's what you've been talking about quite a lot but probably just some of your own code i guess or like kind of like code snippets that are flying around i guess that's my interpretation of the question please let me know on the youtube channel if that's incorrect yes so i mean what i was talking about in the session was the two main ways if you've got c code you either port it convert it directly or you um you call into it i guess they might be asking about um managed c plus i said earlier on you could write you can you still can if they change the name of it to cli c something or other um you still can write.net applications in c and therefore if you had some c code and you wanted to use it in net you can combine the two worlds but i i found it just very complicated to work with um and not particularly straightforward because you're still having to think about how do i go between these two worlds of managed and unmanaged um so i i generally prefer if i've got some c code compile it to a dll if i'm running on a platform where i can do that so one example was um have you any of you heard of the sound touch library which is um it's a play it's a library that lets you do pitch um pitch shifting and also very speed playback and i thought i don't really want to create my own algorithm for that so i wrote a wrap around sound touch for some it's not actually part of an audio because this was for some commercial work i did but it led me allowed me to have really good uh very speed playback and pitch shifting quite easily all right thanks um we have another question from atsushi eno asking could you actually achieve garbage collection free audio code could things like cr a constrained execution region help i haven't played with constrained execution region um one of the troubles with audio is that you're often playing for a long time and you're going for a a long period of time if you write allocation free code then in theory um the garbage collector will have nothing to do and i think if i understand correctly with constrained regions you're you're kind of instructing.net please don't do a garbage collection during this period one of the one of the problems i kept running into with windows forms which is kind of a way of writing windows desktop applications is that um with windows forms whenever you minimize the window that's when the net framework thought oh now would be a good time to do a garbage collect because you've minimized the window clearly you don't care about performance at this moment and so we'd often do a garbage collection when you minimized but that was really annoying for people who are playing audio because they were still listening to the audio and then a sudden a little stutter um and i never really got a good solution to that um dot net just did it and it was very hard to persuade it not to um but maybe maybe the constrained execution reasons would help with that kind of thing all right thanks um we have a few more questions um so ali somay asks is there possibility to explicitly instruct garbage collector to ignore certain addresses or ranges i think that's kind of exactly the thing that the last question was saying with these um the way to do it sorry the way to do it is to just not use dot um dot net to allocate the memory ask the operating system for some memory and the dot net garbage collector won't know about it won't touch it so that's generally what i would do for that scenario right thanks so um then there's coming back to the uh licensing question in the beginning so ivan s has i guess a bit of a similar question um and they're asking what are the licensing considerations and moral considerations if someone wanted to port algorithms or steal code from the n audio repository well because it's a commercially friendly license you can you can take all of the code from an audio you can put it into a closed source product and you can sell that product and it's it's fine you're allowed to do that and so i don't stop you doing it that's kind of one of the one of the natures of open source code um sometimes people have tried to contribute things to n audio that aren't compatible with its license i've had to say no sorry that can't come in um there have been other open source projects that have just kind of lifted and shifted bit large chunks of n-audio into their own projects i feel like it'd be polite for them to to say thanks if they do that sometimes they do sometimes they don't but yeah that's the nature of a permissive license you're permitting people to do what they want and if they want to make money from it they can so actually you said you're using the mit license now which if i remember correctly actually requires someone who uses that in a closed source project to at least uh you know include the copyright notice and that license in their product knowledge that they have used that software so it's not all they have today yeah so it's not just polite but it's actually a license requirement but they don't really have to do anything beyond that do they yeah all right um so eugene jackson are asking uh can you speak a bit more about the exclusive mode versus the shared one i guess they mean like wasabi does that be exclusive what happens under the hood is the question okay so the difference between exclusive and shared first of all if you do exclusive mode you get to pick the sample rate um if you do shared mode you have to work with the sample rate that the sound card is currently at which will either in windows it will almost always be either 44.1 or 48 kilohertz depending on whether the last thing you did was watched a movie or played an mp3 in exclusive mode you it also bypasses i believe that in the kind of internal pipelines of wazapi there are certain audio effects that it can run it can do enhancements on your audio and if you choose um exclusive mode you're asking to bypass some of that so it should be faster there are some other flags that you can pass to wizappi to ask um for even lower latency um which are the flags that i believe that digital audio workstation um authors will try and set to get the you know i want the the lowest level possible but one thing they did do with wasapi was they moved into user mode so it's not in it's not the code the mixing engine is in user mode not kernel mode it means that a bug in your audio pipeline can't take down the whole of windows if you remember back in the day you could you could crash the whole of windows much more easily with with audio applications it's harder to do that now it's less common for that to happen but there is potentially a slight performance overhead because of that so i've never i've never achieved with wazapi latencies as low as i can get with azio um asio seems to be on almost always all cases i don't know whether your experiences have been any different from mine but um wizappi is is fairly low latency but not the most low latency right um thank you um a couple more questions ivan s is asking what is the theoretical maximum volume in an audio when using float is it one yes so a typical way of representing audio in floating point is when you convert it from pcm so a pcm sample goes from you know plus 32 000 to minus 32 000 you convert that into the range plus one to minus one and i'm sure almost all of you who've done audio programming will know that's kind of a convention what that means is you can mix other audio streams in and the volume level can go above one it doesn't matter in your mixing engine but then at the point that you give the audio to the sound card the sound card is not actually operating in floating point the digital audio converters and dacs in your sound card are not accepting floating point they're accepting integer samples and with with low level um drivers like asio you've converted back you've converted back to either 16 bit or 24 bit before you've given the audio to the sound card um and so at that point you need to get back into the range plus or minus one so either you just reduce the volume with a multiplier or you'd use a limiter effect to say i'm going to clip anything that's gone above plus or minus one before i go down to 16 or 24 bit to give it to the sound card yeah thanks that makes a lot of sense there was a comment by atsushi eno on the youtube chat which wasn't strictly a question but i'm just going to go ahead and turn it into a question because i like the comment so they're saying if you convert shout float and buy a race like be careful with indian-ness that might bite on some arm processors and i guess i'm going to just turn it into a question and ask like would some of the manipulation stuff that you showed like should you be aware of endingness and is that a problem when you run on like different kinds of devices yeah absolutely endianness matters um fortunately on windows you're almost always um in little endian and um the times you have to worry about it is aiff files use big endian some azio drivers so azio has a whole long list of supported bit formats it can be 16 bit 20 bit 24 bits 24 bit pushed to this end of the 32-bit word or that end of the 32-bit word and some of the some of them are um opposite endian however i haven't supported that in an audio because i i haven't encountered a sound card that's needed it so there are a few places and the other thing that i can think of is the universal windows platform does run on raspberry pi and i actually got an audio and some audio code running on the raspberry pi and again your endianness is the other way around on that although i was using the audio graph api for that so i didn't have to do the byte swapping myself so the answer is you rarely have to worry about it but there are a few edge cases where you do need to think about it all right um great and then actually the last question here um as far as i can see is going a little bit outside of the scope of your your talk um i guess so it's a question again from monty music channel uh whether there is a good source for the binary layout of a midi file for the layout of a midi file um i can't remember what the website was because it was many years ago i did it but there's very good documentation you can read the full midi specs and find them um on the internet um it's a fair it's amazing how long the midi protocol has has lasted basically um it's been more or less unchanged since the 1980s although midi 2 is literally just upon us with more capabilities um so i haven't looked into midi 2 but if you just search for the midi file format you'll find all of the kind of midi there's a lot of bit manipulation you need to do with midi it's a little bit of a fiddly format to work with and you also need to learn various quirks like with midi even though there's a note off command it's more common to find a note on command with a velocity of zero than it is to find a um an actual note off command um but it's it's a fairly simple file format to work with unless you've got want to get into sys x's and of course when you do with sysexes then every synthesizer has got its own quirky sub format i've been trying to learn i've got a i've got a yamaha modx here and i've been trying to learn the sysex format of that because i want to do some of my own controlling events using web midi so that's what i'm doing for fun in my spare time but yeah just search for midi uh file format and you'll find good um documentation that gives you all you need to know for all of the midi standard midi commands yeah because i do remember like the you know the midi spec with all different midi messages from back i don't know which year it was when when i was actually working in juice team and implementing the mpe spec there right but i have never dealt with like midi files i guess this is maybe what they were they meant like yeah the midi file is pretty much the same it's just um i think there is a header oh yeah so it's all coming back to me now there's type 1 and type 0 midi files yeah exactly i have to know the headers of the both of them i heard this somewhere 10 years ago yeah yeah um but i've done all the i've done if you're a net programmer i've done all of that hard work for you and you can you can convert between type 1 and type 0 and so on all right well thank you so much um i guess that concludes our uh q a thank you so much for patiently asking uh sorry patiently answering all the questions that the people on the stream were asking um and yeah thank you again for this amazing talk i definitely learned a lot and i'm sure everyone else did as well um so yeah uh just yeah thanks again and uh so much been a real pleasure to be with you and looking forward to the rest of the evening yeah me too and with that i'm gonna pass over to uh josh who's gonna tell us what's gonna happen next yes thank you very much timor thank you very much mark for your talk very enlightening we've been doing the rounds of low-level apis we have somebody give a linux api talk a couple months ago and now you've done uh windows now we just need core audio so if you're a core audio expert and you're out there and you'd like to give a talk and take us on a whistle stop tour of core audio please drop us a line or if you want to give a talk of any type please get in touch with us on the theaudioprogrammer.com forward slash submit um and if nobody else gives a core audio talk then we might have to uh twist teamer's arm into uh the giving one for us so um yes great so we're going to move on to the next talk with chris santoro and uh chris is the engineering lead at sound toys in burlington vermont and he's been developing audio effects plugins for over a decade uh he holds a bachelor's degree in computer engineering from the university of florida and a master's degree in music engineering from the university of miami and yes thank you very much for joining us chris it's uh it's great to see you and um yes hope you're having a good time in um sunny vermont yeah yeah we've had a couple sunny days here recently but um yeah it's uh the average temperature temperature has been about 20 fahrenheit and so it's uh that time in the winter so yeah yeah so it's uh there's so much that we could talk about i guess i guess maybe one place where we could start is kind of the beginning uh what what got you into audio software development and curious about wanting to develop audio plugins um well i think my story is not too different from probably most people's story um most people start out getting interested in music right and in uh like elementary school we would do like group reading in class and i really hated having to listen to people read aloud like i really didn't like it and so one way you could get out of that was um to go do violin at that time so i said oh that sounds cool so i learned to play violin in elementary school and then um kind of loved it and um started moved over to cello and uh music is kind of something that stuck with me my whole life but um by the time i was in high school i started to get interested in computers and um some of that is through um you know through video games um and uh really interested in um multimedia technology and how computers and computer technology can um you know create these um great creative experiences um and uh the first doll that i used was reason um and it blew my mind i thought it was so cool um uh you know using that in the you know late 90s maybe around the year 2000 somewhere around there um and so by the time i went to college i knew i kind of wanted to work with like you know multimedia technology signal processing and actually at first i was going to go for a degree called um digital arts and sciences which is kind of like a half art degree but then i decided um i was also attracted to like knowing how things work down to the most fundamental level so i decided to study computer hardware um because it's that's just always been something about the way i approach things it's like i need to know it all the way down to you know almost the atomic level uh there can't be like any mysteries in the stack so um did computer hardware and electronics at the university of florida and then kind of touched a bit on signal processing there and i had a job in a lab where i worked on radar so a little embedded system texas instruments dsp um and we did we we wrote algorithms that processed radar signals for like defense stuff and um that's pretty neat but my defense wasn't like kind of where i wanted to be i wanted to be in something a little bit more artistic and creative and uh so for our senior design project um i did a guitar pedal like on an analog devices um development board with a friend of mine who was also interested in audio and he was from the miami area and he told me about the program at the university of miami uh music engineering technology and um so i decided to apply and uh i went there and it was it was great it was um it's in the music school it's not in the engineering school and that was something that appealed a lot to me um it uh and so you're around musicians and music 24 7 and around people who are interested in that all the time and so it was really great to be immersed in that um and then um after miami i did an internship at sure um the microphone company in chicago that was really great and um the then the economy collapsed and so i think had that not happened i probably would have ended up getting hired at sure uh they wanted to bring me back as an intern again and um because they didn't have a job break at that time and um then i heard through the grapevine from somebody who knew about soundtoys about the opportunity there and um it sounded really cool because you know going back to my experiences with reason um making plugins was always something that like would have been my kind of like my dream job so um i got hired and i've been there since you know 2009 so it's been a fun a fun ride that's great and uh so you said that you went to the university of miami and we've had so many people or i've i've met so many people through my travels that have uh that are alumni from there we have brett porter who gave a talk last year on midi 2 spec jay coggin who used to work at apple he's a amazing uh engineer uh and there are loads of others that uh i can't think of off the top of my head uh but you have the legendary will perkle uh who was there he he was one of your professors there at uh miami is that is that right actually um that's not right i was i was there during the non-will purple years so like will was there like in the 90s and then he went off and worked at korg for a while and so late 2000's i was there i um the professors there were this guy named corey chang who came from dolby labs um really brilliant guy in colby lighter who is kind of an electro acoustic music um uh expert and um really interesting and fun guy he works at magic leap now um so then after and then i was there for just for a graduate degree so it's just two years and so will kind of came back after that um and i'm good friends with will but i don't know him from having ever been his student i just know him through that community yeah amazing yes so many people so many great engineers have come have come from miami it's uh it sounds like they have something in the water there for sure you know i think what it is is it's it's very um industry connected it's not exceedingly academic um and so everything's built around learning things that you can use to make real stuff and to you know do things that are actually useful rather than learning theoretical concepts from people who have you know never worked in the industry like everybody there has has you know real world experience i mean and will's a great person of that like through his work at korg and you know a lot of other stuff that he's done yeah amazing and big shout out to uh christopher bennett who's a professor there as well i'm not sure if you know chris or not oh i know chris very well yeah we're both kind of from the same part of florida um so and i was there when he was getting his phd so yeah we're good friends that's great yeah chris just came out with a book recently uh i can't remember the title of it off the top of my head but uh it's a great book i uh he actually gave me an advanced copy so go out and get it it's dsp and matlab uh really good book uh so uh moving on so your uh your time at sound toy so you joined there in about 2007 you said and um you were telling us before the call that soundtoys was among one of the first plug-in companies that uh ever really came out with with audio plug-ins so uh of course you have probably even even at the time that you got there the code base must have been um you know over 10 years old so can you talk a little bit about the challenges that you had to deal with uh working on a legacy code base and uh that would have been pre-juice as well so uh another thing is that the plug-ins that you create there are are not created in juice they're outside of juice so can you just talk about some of the challenges that you have to deal with uh working outside of a cross-platform framework well we actually work within a cross-platform framework but it's one we've built ourselves so like like tying back into what i was saying earlier it's kind of a good match for me like soundchoice was a good fit for me because um we like to understand everything top to bottom and control everything top to bottom so the all the advantages of vertical integration right is if you want to do some custom you know stuff that might be really weird and not apply to a whole lot of people um you can do it you know you're not really tied up you're not limited or waiting on like any third party to implement that if you're using some kind of third-party thing um and the disadvantage is you know you have to do the work like you have to implement all those details yourself and all the um you know all the neat stuff that the the juice team takes care of for you um i think if you know we've talked about in the office like if soundtracks was starting out today like we'd probably use juice i don't know i think it it makes a lot of sense um but we've always been in the completely 100 proprietary boat and so that's where i think we're gonna stay because you know once you're in that lane it's it's more work than it's worth to to get out of it but um yeah so i mean talking about a long-lived code base like um soundtoys was one of the first plug-in companies um writing plug-ins for pro tools in the mid-90s so and those plugins ran on you know you didn't really even have um native plugins that ran on your computer they ran on accelerator cards so there was the pro tools tdm format which was a fixed point um motorola dsp um uh that only high-end pros had you know so all the people buying those plug-ins were like people in big studios that made a lot of money you know mixing records making records usually um and not like you see today where you know you have so many people buying plugins and making music at home and they you know they might be doing it professionally or they might be doing it for fun um but um yeah i think like going back to your question um the challenge the main challenges were you know a lot of our framework was tied to um being able to support the pro tools tdm format and native code because we only implemented vsc and audio units yeah around 2007 i think um before that it was you know pro tools was the only place you could get um our plugins um and so decoupling from the limits um there were certain limitations that were kind of built into our code based on having the capability to run on hardware um things like memory limitations and how we allocated memory so um when we did a we have a product called effect rack which is all of our plugins built into one it's kind of a meta like multi-effects plug-in and when we built that um it strained a lot of those um you know those limits like like like having a maximum amount of memory that could be allocated um because all of our memory is allocated in like a basically a one memory pool you know for different types of things and that and that was related to um the this motorola chip and it had different it had different types of memory that um you would use for different types of things like signals or data or um that kind of stuff so that has some side effects that we have control over that like you get really good cache locality out of your you know your data structures and all of your stuff that you're doing in dsp rather than if you were just allocating dsp data for everything separately in every one of your algorithms where there might be all over the you know uh the place but um yeah so i think yeah it's things like that like um having to have a dynamic you know converting to a dynamic memory pool um and um yeah that's i think basically it things of that nature without going into it too much yeah um yeah i actually can i ask you a question or maybe just drill down into this a little bit more because i'm just really curious you know how a lot of companies like native instruments or ableton or others they actually do the same thing they have like their own cross platform frameworks that they have written before jews was around that they kind of committed to now and they keep maintaining and i wonder how it is now like like 2021 working with a code base like that like where you know a lot of people are talking about like the newest c plus plus standard and all the like fancy new features that we have there or a lot of people are talking about things like unit testing we had you know several talks about that at the last adc and i know that these and other things can be a real challenge if you're dealing with a 20 year old code base um you know because i've also worked with some of these these places and so i'm really curious like what's the experience like for you uh what's the experience like for that with respect to um uh people talking about like hot new hot new stuff like uh well like okay let me be a bit more besides like whether you kind of can or do like embrace these like new like you know current language standards or like things like unit testing or whether you kind of um have to like just follow like a different strategy just because of like the code base forces you to do that um i think it's it's generally pretty flexible i mean i think one of the things that is might be like a shock to newer developers coming in is um you know our code is is mostly c plus just like you know almost everyone else but um c plus plus even means different things to different people depending on how old you are right so like ours is the older is basically mostly pre c plus plus 11 c plus which is to say like it's mostly c with classes um and so it's very c-oriented because um the founder of south toys ken bogdanovich and um another long-time engineer at some toy spot belcher um both of those guys came from eventide in the 80s and 90s um when you didn't even necessarily have c plus plus um they might have been working just with c back then so our our code base i would say is very c oriented and there's no there's almost no standard library in it because um standard library used to be slow and it used to have it used to be hard to debug the tools have since c plus plus 11 it seems to me like everything's gotten a lot better and um you can do a lot more at that stuff now but um kind of just because of convention like we have avoided um standard library in a lot of our code um and so we we do a lot of rolling our own you know for for things like that um so that's adding two things like you know threads and strings and like you know all of these like basics yeah it does include um strings um uh not for threads we have a little bit of the um code that we've done that has involved threads and concurrency like we've used a little bit of um um threads in unixes and things like that yeah this is a practice that i've actually heard happens in some gaming companies as well that where uh they feel that the standard library is actually too slow to uh for their own use and so they roll out their own standard library that's slow and sometimes it's just not offering like the right guarantees you know where like you have like the standard library being optimized for like you know a certain algorithm being like oh log n or whatever but what they need is like something something different from that just because of their use case so yeah i guess i think i think like big gaming studios like uh what is it like ea or whatever they're called like i think they all have like their own implementations of the standard night which are just like a tiny little bit different you know yeah and i think a lot of like the old like audio companies they have something similar going on you know the ones that come from the like early 90s or like early 2000s like i think i think they probably all have the same thing going on if they ship like on multiple platforms i would say yeah that's exactly right yeah i think in you're also right in that it has the most similarity i think to the the game world um where you're running on uh and especially again especially the older audio companies or the ones that have been around longer like um the have more often would have run on esoteric platforms you know like um that where you didn't really have nice um native environments like you do now yeah have you have you ever tried to uh create something with juice have you ever tried juice out just in your own free time um i had there's it's been on my list to do i i'm not really that familiar with juice to you know to be honest um but it's been on my list to like take a look at it just to kind of know i think what like um younger engineers you know they're most likely to be familiar with juice if they're gonna start their own plug-in projects so to know what like uh what their expectations are coming in like whatever what type of environment are they do they think they're going to see you know um but yeah yeah that was going to be my next question i imagine that having a code base like that that there must be certain challenges for hiring uh hiring engineers and what they would expect to see have you have you found that in your hiring when you when you go to look for other engineers or do you feel that you have an infrastructure where you can kind of ease them into this code base um we haven't um we don't hire very frequently we're very uh been a very small company so um you know we're about 14 people and most of that is you know probably customer support and um the dev team is four or five um people um so you know we're not i don't think again it's not my company it's um but uh the way i my take on it is we're not trying to become like a native instruments or waves or something like that where we have a hundred people or you know more so um hiring is very infrequent um so i yeah i wouldn't say that's really a it's never been a problem i think like um anyone coming in as long as they know c and c plus plus like um you can get into it and there's someone there with a lot of experience to guide you you know to get used to that environment yeah that that brings a great brings up a great point which is about the company philosophy so you've so this is a company that started up in the mid-90s and that at the moment you have 14 people in the company and it's interesting because a lot of companies have started then have grown into kind of behemoths you know your arturias and native instruments and can you talk a little bit about what the company philosophy or what the company outlook is uh is it is there a desire to stay small and kind of more of like a boutique uh plug-in company what uh what is the kind of company philosophy there uh yeah i don't know that the there's necessarily like a desire to say stay small but i think that the desire is like definitely some things i've heard um you know can the founders say is is you know there's a desire to be sustainable um so if you are a company that's chasing like always growing profits like that's not sustainable that's also something i've heard the ceo of ableton say which was really nice at adc uh last year his talk was really great um and um yeah i think sustainability and doing um meaningful you know meaningful work so not like cranking out products just because you know you need a new product like cranking out something because you think it's unique and new and fun and is gonna make a dent in the in the music industry and um yeah have an effect on you know how music sounds yeah it's a lot definitely yeah i think that's a really uh really healthy approach i would say yeah and we've i will say we like we've been fortunate to have been successful enough um that we can take our time with things and it can be like it can be a little frustrating sometimes like when because you you know as a developer you want to release products right um but um the bar is is really high for us so you know we work on some things for a long time and and we even work on things that uh a fair number of things that don't go out the door because we uh you know they just don't they don't meet the bar for us or they sounded like a good idea and then they you know it kind of wasn't um you know just didn't meet our quality level that we were looking for for you know yeah and can you can you define what that what that is the characteristic of what makes the soundtoys plug-in uh the because of course uh many people use it use them and know them for their unique concepts unique ideas can you describe where that where that stems from yeah i think uh well that all comes from ken um he's like the driving force behind all that so um and he's that's i think one of the things i like most about working here is he's got an encyclopedic knowledge of effects um soundtoys also has like every an incredible gear collection that i think like nobody else has like i don't know if you've followed us on instagram or or seen things on the internet but like um we have it's the noah's ark of gear so like every effects box that's like ever been invented is pretty much in the soundtoys basement like it's like so anything you want to check out to get inspired is there you can go use it for real um and so i think um to answer your question though the uh the what makes it sound what makes sound toast plugins unique is they are easy to use um but also deep so it's this balance between um you know not throwing a thousand knobs on the screen um but having uh an interface that can um that encourages you to play with it and kind of like kind of um expose new features where you can go very deep if you want to um but also i think um making plugins that are hard to make a bad sound with so um you know that no matter what you do you know you're going to get something that sounds good and interesting um yeah so i think that's kind of the that's the core thing to me is like balance of ease of use um fun to use um and but also like incredibly tweakable like where you can go down to you know and really go out there if you want to go there yeah when i was when i was producing music echo boy was probably one of my top three plug-ins uh and for that reason because when you look at the interface initially it's it has just a couple knobs that you can just tweak and then if you really want to go deep you can go the the interface leads you deeper and deeper into different types of delays and you can actually customize those delays you can separate the delay of the right side and the left side i i almost felt that that delay with that delay that was the only one that i needed i never needed another delay because it could make pretty much absolutely any sound right i think and that's our that is um the maybe the like prime example of like the soundtoys design philosophy that's our goal like almost with every plug-in would be to to be like that to be the one thing that you need of that of that type you know um to very thoroughly explore that you know that area of effects so um yeah but you know it's it's very it takes a lot of work to do that it takes years like i think echo boy i think took ken like four years to make so um it's uh if you want to make stuff on that level like you can't be cranking it out every year you know um you can't crank out a product every year if you want to do products like that you have to take your time um so yeah and does does ken still code or does he really provide the vision and leave the coding to the rest of the team oh he definitely stole codes um the um uh i don't think he'd be doing it um if he wasn't still like helping create the stuff um so yeah he's definitely you know still very involved in the technical aspect of stuff amazing uh so you brought up that you have a lot of gear there at uh at the headquarters and one thing that i was wondering about is just around the subject of dsp algorithms themselves uh do sound toys as a company do you get involved in modeling analog gear or do you do you simply use it for inspiration do you try to actively do like white box modeling or do you just uh try to maybe approximate sound uh what if as much as you could tell i know there's some secret sauce there that you probably can't tell us but yeah yeah i think what i can say is is definitely both of those things so like the two things you said is analog modeling like yes we do do that and um and just simply using things for inspiration um you know we don't want to be seen as like a modeling company like um if we did like we could have probably done like 10 or 20 model dqs you know like in any time we wanted it would be you know but um you know we we have restrained from doing that um and um and we all anytime we do something we want to we want to model it to make sure we get the sound right but um but then also you want to take it further and add something new you know to the conversation um yeah and what what do you feel makes a because of course you know now you have some dsp algorithms that you can find on around the around the interwebs uh and uh of course through books like will purkle's book eric tarz got a great dsp book of algorithms as well and of course we have these kind of standard i guess you call them dsp algorithms for making a compressor or for making a filter what is it that you feel how do you get into the interesting space how do you get out of this kind of standard so there would be like a standard space i guess for making a delay or making um you know making a filter i guess how how do you get outside of what the norm is is it just through experimentation or what do you use to actually kind of fire those creative juices as a dsp engineer um i mean i think listening to stuff is first is you know just finding things that you you love the sound of and then drilling down into like why does it sound that way like what makes this compressor or this delay or this you know electro acoustic thing like what makes it sound the way it does and then you go about trying to recreate that sound like however you need to um the uh yeah and i think another thing about like how to get outside the box is to um not maybe like don't look at that stuff like if you know there's a if you know there's a there's other off the shelf compressor or filter designs like you know maybe just try and do it without looking at that stuff and you will naturally come up with something different um if you don't let yourself be influenced by you know what's already there now that being said there's a there is good reason to study what people have done before you because they probably had good reasons you know but um but that's just that's just one thing is to not you know just start blazing your own path you know rather than um yeah and and in terms of being a dsp engineer so many people ask us all the time what type of math would i need to study to really be an effective dsp engineer i think differential equations is like you know maybe one of the biggest ones linear algebra also very huge and but i think other than just uh own numerical analysis also um other than um getting like out of the math realm something i like to see in people is people who understand how computers work at a very low level so like not just being a computer scientist that understands how code works but understanding bytes and understanding fixed point floating point dsp and understanding like how a floating point number is represented and understanding the implications of like a fixed point um environment on your algorithm and on filter stability and those things are have kind of disappeared like fixed point processing um and so you know and going back to what we're talking about earlier like with our code base like a lot of our algorithms in filtering technology like were you know um had to simultaneously run in floating and fixed point environments um so there were things about the dsp code that you know um were due to um you know fixed point um so yeah i think in addition to math um you know really knowing deeply how how computers work and what happens to sound when it goes into your computer gets mangled up by the processor and comes out the other end amazing yeah i think i'm i think i'm out of questions uh yeah don't worry we have plenty of questions from the youtube chat great right on time uh-oh uh should i should i start yeah okay so uh we have actually two questions from cameron minuti um and i guess those questions should be prefixed with um you know if if you can at all say anything about this which you might not be able to um so the first question would be um how does santos model uh dynamic behavior in analog circuits what sort of dynamic behavior is occurring in the decapitator ah okay so um yeah this is gonna be one that i can't answer too much um but the um i could tell you what you know decapitator is based on um um you know a bunch of tube amplifiers and i can tell you some of the dynamic things that happen in tubes when you when you drive them really hard which is when you hit a tube you know really really hard it actually um kind of compresses the signal the the transfer function of the tube um changes you know as you hit it so like um you it you'll kind of get um a dynamic a dynamic distortion the character of the distortion changes based on the past input level of the signal and that has to do with the circuit and how the capacitors around it um uh basically like charge up and change the bias points of the um you know of the tube or transistor or whatever the amplifying device is so um yeah that's um yes definitely dynamic behavior i can't go into specifics of like you know how we modeled it um i will say you know the thing we always say is that listening is the most important thing um in modeling uh and rather than like what type of technique you use or or what the actual dsp is but um but yeah there's um i sorry i can't go into more detail there yeah so i guess that's going to be the answer of the second question as well which is does santos use space state or wdf modeling methods what was the second thing say state space and what space or wdf oh wait wait digital filters um all right yeah yeah uh no no we don't use any um way digital filter stuff very familiar with it um it's it's super interesting um some of it's over my head even that's uh but yeah we don't use any of that i've experimented with some state space stuff not in filters but in some other things that i've experimented with so yeah i mean state space is comes in big in um if you wanted to do very like literal circuit modeling you know all right well thanks for for some insight there um bon sembianje is asking what kinds of tools and knowledge what kind of tools and knowledge gives you a degree like yours in music engineering i guess the question is like uh like you know you said you had if you have a degree in music engineering what like what are the kind of tools and knowledge that you get out of it which you can then can apply to the kind of stuff that you're you're doing i guess that's that's how i would interpret that yeah so um i think uh well signal processing for one um electronics um acoustics um psychoacoustics which is how your brain interprets you know sounds um and and then that is all blended with um you know basic computer engineering stuff too and that's kind of like how i would describe the music engineering degree and then also it's very focused on the recording and music world all right um thank you um we have a question from monta music channel um they got the full package of your plugins and they wonder if there is a course of flanger unit planned uh not phase mistress microshifter or crystallizer oh okay they're asking about a flanger about a chorus or flanger yes uh um uh yes yeah i'll just say yes okay yeah we have to if you search probably way back to like the mid 2000s i think we might have even like um advertise the flanger plug-in that was in the works at some point but it's definitely something we want to come back to and and chorus also for sure is like would be in our wheelhouse it's something we've been talking about for years all right um interesting question from bonsambiante how do you deal with the sound design um are there music producers in the team and can you as a developer uh like raise ideas about the sound aesthetic uh yes absolutely so like i think um kind of touching back on some stuff josh was asking about about um hiring people i think one of the biggest things we look for is are people that are you know passionate about music and care about the the aesthetic um and care about design um and um those sort of things and um so everyone on our team like has that perspective where they are they have an opinion um which is great and then also like difficult sometimes because we don't always all agree but um the um uh yeah i think um we our main sound designer is this guy named mitch thomas who is um really a big synth head he knows the entire history of synths he's really talented and he's a great preset designer and he kind of leads like most of our um you know preset creation but but we all chip into that and that's also one of my favorite things about working for a small company is that you do get to have a say in that sort of thing and you get to have an impact on you know on sound quality features you know um all of that that sounds like a very good place for me uh to me for someone you know someone who's a developer who also cares about sound and all of these other aspects of like what it takes to make a plugin so sounds yeah yeah but there's a drawback too which is um you don't get to focus on one thing all the time like sometimes like i i fantasize about like oh it would be cool to work at a place maybe like uh like native instruments or something where i could be a dsp engineer and just do dsp for like a couple years you know but uh so it's um it's a plus and a minus like you know getting to do everything means you kind of also have to do everything you can't really like stay in one little box you know that being said like i know a company which is not native instruments but another another big one where they don't really have people who just do dsp and at the same time i know like a very small company where they have someone who is doing just dsp so it's not just the company size i think it's it's more like the team setup the team culture things like that so you know like the initial people who like started it and how they approach it and how it kind of grows from there so i don't think it's necessarily the company size and that kind of determines what your setup is going to be i think it's more like you know the kind of the culture or like the individual people involved or i don't know but like that would just be my guess yeah i'd agree with that i think that's true um so we have one more question from uh rebel uh talagka um who's asking um since you worked at sure as an intern the microphone company could you mention any skills that companies like that look for say in an undergrad electronics student um yeah i think they uh it's hard for me to speak for them for like what they look for i don't i don't know but um you know i think they probably just generally want you to be um pretty smart and they were so at the time that i did an internship there they were just um getting into building a dsp team for wireless mic stuff because that was around the time that um the big like wireless spectrum changeover was happening and there was going to be a big disruption in like wireless microphone stuff so they were looking at using dsp to kind of you know solve some of those problems so um yeah so i can only really speak from that perspective like um yeah just to to know the fundamentals of dsp very well um and you know be generally um pretty smart to know a good bit about embedded um embedded systems and actual ds writing code for actual dsps because they make real physical products um in the world that are going to have like uh analog devices or texas instruments dsp or something like that in them um yeah so yeah i could chime in a little bit on that as well because i speak to a lot of companies about hiring and what and what they look for i think what they look for more than anything is just somebody who's willing to learn new things not afraid to get their hands dirty in terms of getting into code learning new things being able to take something that they don't that that the developer doesn't know about and break it down into digestible pieces of information that uh and and kind of working their way to a solution um and also building stuff uh there's i i can't believe how many uh people come across uh that i come across that want to uh be plug-in developers and companies say well show us something that you've created show us something that you've made and you say well i haven't created anything i haven't made any i haven't made any plugins i don't have any code that i can show you they need to have something that they can see that you've created something that shows your ideas and you know your your way of thinking yeah i was just about to comment on that it's a little bit of a chicken and egg problem with many people isn't it um it is it is a little bit but i think i mean with the tutorials that we have we have what over 80 tutorials 80 90 tutorials now on how to start building your own plugin uh and i think there's enough there to really kind of get started with something even if it's just a delay even if it's just a standard delay and you've and you've done that and then you've created your your own ui with it just a basic ui i think even just having that just says something about your your mentality as a developer you know whether you like to uh just have everything just kind of all in the process blocker whether you'd like to write a little function write functions for everything i think it just really says something about the way that you process problems and really think about them um yeah and i guess also your choice of like do you use juice do you use something else do you use like modern t-plus plus or not like how do you structure your code and things like that right there's a lot of stuff to look at if you're like hiring from the other perspective you know from the other side of the table yeah they don't need they don't need like the mona lisa of plugins you know that's not especially when you're talking about a junior developer you know the the bar in terms of what's expected isn't necessarily um you know they're not going to expect something that they would expect from somebody like chris who's been in the business for a long time but it's just showing that you've made something uh and that you have an understanding of the basic fundamentals and by basic fundamentals meaning a lot of stuff that we cover on the channel like uh you know are you trying to call call to your ui from your from your uh from your processor and things like that separation of data and ui and just basic kind of fundamentals of not allocating in your audio callback those are the things that they look for and like i said curiosity to learn more than anything just curiosity to digest new information i would also and we are kind of derailing the conversation completely but i think it's maybe just interesting for for people because these questions can't come like up every single time right we have this meetup where like people are like how do i how do i get a job or what are you looking for in this kind of stuff so i think uh probably there's also like a level of like cross-pollination if you have like other background which is kind of not necessarily building an audio plug-in but there's some overlap with that i think that's also like really useful because for example speaking about myself when i got my first job as a software developer at native instruments which was when 10 years ago now um i had certainly not written a plugin before i i was doing like astrophysics like numerical simulations which is something completely different but i knew some c plus plus and i knew some in some other contexts where you have to like churn numbers and do things like that and i didn't really care about music and i was really interested and i think so that was for me enough to get that job and i think it's it's just you know for other people as well maybe interesting that there are other ways to get into it if you have like other you know relevant knowledge which kind of touches upon audio which you can reuse maybe and you just kind of show that passion yeah also kind of a way into the industry i would say or at least it used to be yeah absolutely there's definitely not a one way to go uh you know like you said like if i if i saw your cv back way back when and i saw that you were an astrophysicist and you knew some c plus i would probably think this person probably can learn on the job like coding plug-ins will probably be a breeze it would probably be a breeze to uh to somebody that could do astrophysics and uh but people have all kinds of interesting backgrounds like one of your degree in history and you know he's an amazing developer right yeah yeah yeah absolutely and and uh and music as well sometimes just understanding what the user wants i i mean i don't know i think it would be quite difficult to be an effective plug-in developer if you didn't actually produce music i mean i know that there are people out there that do it but uh i think that uh there's definitely a certain experience that you get from creating music being a musician you have to have i'm a strong believer that you probably have to have some sort of base knowledge or base interest even in uh in music how music works uh music production um you know yeah i think that's the uniting factor right so you know chris you mentioned you learned the violin and then the cello and then mark i see like a beautiful prs guitar behind you and you know i also was you know when i was applying for that job i had spent like years making music and playing in bands and you know playing around with guitar rig which is the reason why i wanted to go to ni right and things like that so i think this is maybe like a uniting force that you're all just really yeah like music and audio and sound and then you know yeah and not to segue too far from the conversation but um another another thing another common question that we get is well what do i build you know what what do i build and uh one way that just kind of catapulting off of what teamer was saying is that if you played the violin for instance you would probably have some some type of intimate knowledge about how strings work how strings resonate how about music theory so use that use what use that experience that you have whether it's physics or whether it's uh whether it's playing an instrument or whether it's producing music and build try to build something around that try to build something around that base knowledge that you have and because you're going to likely know more about that particular side of uh side of things than the average person so that that would be the place to start like i know more about djing than the average person for sure so one of the first apps that i built was piece of dj software uh so you know how the how a person would want to use that software and so on so that's kind of that was kind of my starting point um yeah so anyways let's get back to chris yeah yeah we have two more questions for you chris well if i want to make a couple comments on what yeah yeah just to like reinforce what josh said about about building something i would say it's also like don't get hung up on you know like when i'm interviewing somebody or looking for somebody um i almost don't really care what like technical choices they made i mean i do care a little bit if they made really bad technical choices that's a red flag but um the the fact that you chose to build something is what matters most like it doesn't really matter the details didn't really matter the fact that you took the initiative to build something and then also you know about what to build um yeah it's just like well what are you interested in like what do you want like you know and that will come through on its own so if you say like oh man like um like when i was at miami um i made a synth because i loved synths and like it just it was just fun it was just what i was interested in so i just wanted to know how it worked i wanted to see how it would sound if i tried to make one so yeah great um i did have one question before we got to the last two questions um that i forgot to ask earlier uh so so soundtoys is a uh is a smaller company and of course uh there are a lot more plug-in companies that are out uh releasing plug-ins now what do you feel that the company has had to had to adjust uh as as a whole in terms of uh saying okay well there are a lot more competitors in this space and now we need to pivot in a certain direction or do you feel that you've estab as a company that you've established that identity and that you can say we are who we are and this is what we do and we're going to continue to bear down this path like for instance have you thought about branching out to let's say ios apps or anything like that or like going in different directions building a virtual instrument or has it always been okay we're going to we're an effects company and that's what we're going to do and that's what we're going to that's our bread and butter uh i think you know as far as ios apps or instruments like we've of course talked about doing all of those things um but in terms of like how the plug-in like market has changed you know over the years and there being so many more people like um i'm not sure it's really affected much um in terms of how we approach things um i don't think it's it really matters because i think everyone has their own unique perspective it only matters if you don't have a unique perspective then it matters that there's a lot of other people around but if you do have a unique perspective then it sort of you know it sort of doesn't matter and i i feel like you know we do have our own unique um take on things in our own unique approach so amazing uh teamer got some more questions for us yeah so so one is i think you've kind of answered in your talk but i'm just gonna for completeness sake uh repeat the question it's from a monty music channel asking did you write your own api on top of the steinbark vst sdk or do you use juice or do you write separate for all plugin formats and i guess what i remember it's kind of neither of those right you have like your own api that you wrote before you actually put vst underneath that right yeah so there's kind of like a platform independent format like an abstraction of like this is a plug-in and then um you have um you know basically wrappers are around that for vst aax and um audio units you know that just translate between like the different ways of like how do you get a parameter value how do you identify a parameter how do you you know declare what parameters are automatable and which ones aren't you know um where does um how does midi come in how do you know all those types of things which are just like all just slightly different between au vst and aax and um and yeah so there's kind of like a core a core center and then there's just like interfaces that you know interpret between you know what a vst host wants to see versus what an audio unit's host wants to see yeah it sounds very similar to like the equivalent library at native instruments that at some point i was the maintainer of when i was there and sounds very similar to kind of what juice is doing in the end i think you just if you want to do cross-platform and cross plug-in formats i guess you just end up kind of writing something like that which is probably going to look fairly similar everywhere at all of those companies i would say probably exactly yeah yeah and then the same thing for you know you're talking about cross-platform like you know um you have a some type of file api you know which has different implementations for mac and wind has a little interpretation for mac and little interpretation for windows well actually since superstar 17 we do have stud file system for that but i guess in the audio industry not many people use that because they already had solutions like a decade before this thing right yeah so yeah that's the trick like if you already have a solution like even if the new solution is great it's more work to move to the new solution only to get what you already had which is working code you know exactly and it's not more just more work but also you have to retest everything you have like yeah without actually getting any value for the end user right exactly exactly yep yeah yeah so i think yeah it sounds like uh it makes a lot of sense what you're saying um right so that brings me to the last question by k cranberry who's asking when will the big plate be released uh i can't answer that question um the uh the again like i'll just point back to like the high bar for things and like uh it'll be ready when it's ready um there's been like a lot of work put into it i think at some companies you know it might have been uh released already but um but we do you know we go to the beat of our own drum and might take things in a different direction than people expect so um but we'll see you know i don't know i can't say how that's all going to play out so well we're going to be um i guess watching carefully and looking forward to it if and when it happens so yeah yeah with that um we don't have any more questions on the chat unless you want to very quickly write something now or let me know that i forgot something uh which doesn't seem to be the case i was i was interested to ask chris um whether you have to do any trade-offs of cpu usage versus your ideal algorithm dsp algorithm that you want to write and how do you how do you make that trade off yeah absolutely you know and i think those trade-offs the the line where the trade-off exists has changed over time which is a challenge right so if you have a um a product that you designed like a long time ago you know like there might have been like certain considerations for cpu that where you might be able to make it sound even a lot better now um yeah we definitely run into that and i think even if you know with computers being so fast you have to consider that you're not the only plug-in in someone's session or they might have 20 copies of you 20 instances you know in their session so um you know optimization is yeah it's still there's still limitations to work with you know yeah great uh yeah it looks like that's all the questions that we have from the chat as well thank you so much chris for for sharing that with us i appreciate it no problem yeah thank you very much that was definitely a very very interesting session thank you so much for like sharing very generously sharing cool yeah you're welcome this is fun great uh okay i think we'll sign off uh now and once again we'd like to give a big thanks to our sponsors uh uh sonic's focusrite and juice for supporting the audio programmer meetup and you can join us next tuesday i don't have the date in front of me at the moment 9th of march and actually we already have at least one speaker whom we already know um i think we're gonna hopefully announce all of that soon yes um but yeah it's gonna be pretty cool i think so um yeah look forward to that uh tuesday the 9th of march at 1800 uk time yes and uh and once again we're always looking for more submissions once again if you're getting started and uh and you're looking to get your start and you've created something and you'd like to present it and uh and show people like chris what you've the amazing thing that you've been working on you can submit for a presentation on the audioprogrammer.com forward slash submit and also let's continue the conversation on the audio programmer discord uh you can come join us on the audio programmer dot com forward slash community and with that we will sign off thank you very much mark and chris for joining us and all that amazing information and thank you to everyone tuning in and asking questions and listening and being part of this thank you so much um yeah and i guess we'll see you next month yes we'll see you on the mark on march 9th all right bye everyone hi everybody cool we're off yes wow that was

Channel: The Audio Programmer

Views: 815

Keywords:

Id: POTZyXnHUCY

Length: 146min 35sec (8795 seconds)

Published: Tue Feb 09 2021