Advanced WebAssembly with Ben Smith

[Music] first off my name is Ben Smith I'm software engineer on the web assembly team at Google and this talk is very much meant to be sort of a follow-on to what Thomas talked about so I called it web assembly to a one porting software to web assembly so Thomas sort of got you into the mode of thinking about like how would you write web assembly code using Him script in and I'm gonna do a little bit more but first I want to take you back to 2010 back in 2010 you may have heard this let's see if it actually works so this sounds probably a little bit different than the Justin Bieber that you're used to because it's slowed down 800 percent a person named Nick Pitts singer made a version of this and the internet kind of went nuts for it like this this YouTube video has 3 million views that's not a lot that's that's Justin and everyone's like wow it's like Justin in a choir you know it's really pretty amazing so people kind of went nuts for this you can sort of see let's see there we go lots of news articles people saying stuff like if you slow down Justin Bieber by a hundred percent you get art and Justin Bieber is amazing when he slowed down to 80% and what they found was there was actually one tool that was kind of behind this all it's the tool called Paul stretch by this person named Paul Nazca so they NPR actually went and talked to them and talked to this person and and said like well how does it work what what is it and it's like oh it's actually pretty simple piece of software I wrote in 2006 part of like my you know I wanted to be able to do audio processing and so on and then it wasn't until 20 into 2010 that people actually found it and played around with it and really found that they liked it so this is the link it's a SourceForge page remember those remember this is 2006 back when when the news was a little bit slower and this is kind of how it works so the idea is that you take this audio form file and what you're sort of looking at here is is sort of the WAV format and it sort of takes these chunks of audio randomly and then it spreads them out and so what's amazing about this is that you can actually slow down audio from 8 X to 10,000 X in fact there was a story about one band that was like well we're gonna release our our album in a year and we're just gonna put up a livestream playing at 40,000 decks and eventually it will finish and then then we'll release the album that's what they did so this is a really really cool piece of software I thought like what am I gonna do for this talk I had no idea and literally two weeks ago I was like let's see if I could port this so this talk will mostly be my porting tips in the future if you want to follow me on Twitter I had it up at the at the beginning I will write up a little blog post with a lot more details but for now you can basically just you know dig in and listen to a few of these tips so tip number one build normally first it seems obvious right a lot of these tips will seem obvious but this is like me porting a lot of code one of the first things that people do is like I'll just try and compile it with my fancy new and scripting compiler see what works see what breaks no no no build it normally first so here's the step that I did first pull down the library Paul nasca slash Paul stretch using get get these nice get output and then I install all the dependencies so you can see quite a few dependencies here and you also see that there's a command to actually compile this normally so this is really really important to do the reason that it's important is that you learn a couple of things about compiling normally what you learn is look at all these libraries that I needed live audio file live Orbis Lib FL TK port audio Lib mad limit MX ml these are all things that maybe or maybe not I will need to port to make this project work okay what all I also learn is oh actually when I compile the code I know that it actually compiles I know that it actually runs you know I got something that I could play around with so great step 2 for C++ using script in now Thomas talked a little bit about this there's lots of other pilers for webassembly one of the big benefits of webassembly is that you can use many different languages the next talk will talk a little bit about using rust but for C++ and scripting is definitely definitely definitely the way you want to go use em script in for that so Thomas mentioned this as well I'm scripted org super easy just download it pull it up it'll give you a compiler that you need this is what that page looks a little bit like it's like I got a lot of great information API references for a lot of the the tools that I'm scripting includes so I'm script in is meant to be giving you a lot of functionality like like STL or WebGL built in like timers built in so it's more than just a compiler like if you know C++ compilers a lot of times you think oh well it's just gonna give you the basics right input-output prints things like that no no this gives you a lot a lot of stuff built in so I'm scripting is really really great for this kind of thing so when I download M scripting the first thing that I do is just test the really really really simple project so all I did here is create a completely empty C file run a m plus plus that's the compiler this is exactly the same command I we didn't we didn't share notes this is just what I did and then just make sure that it actually gets you something so here you can see empty is the date is wrong remember that date everyone that's a pretty important date coming up okay moving along so tip three only port what you need so remember before we had a huge list of libraries we had audio file Lib forbus everybody know OGG Vorbis probably yeah mp3 all these files we could port them well you know having like borkus of course verbis of Corbis so what i do when i'm porting something like an application when you're porting a library a lot of times you can just port it straight when you're porting an application often what I do is try and see which parts of the application do I need right because a lot of what this application has is like code for for generating GUI interfaces and things like that I don't really want that because the web gives me my own GUI interface what I want to do is try and find the parts that are important for what I'm trying to do and then cut out the parts that I don't need so what you can see here is the entire entire contents of the repo and the things that I've bolded here are the things that I sort of just looked at and I went like okay well what sounds like what I want you know I saw no control okay maybe free edit no that doesn't sound right player okay that sounds interesting process stretch well obviously I'm stretching out audio that sounds about right okay stretch so this is kind of what I looked at and actually when I first started porting this I started with process stretch I was like oh that's got to be it and then I looked at and I was like oh actually maybe not so you'll end up having to sort of dig a little bit into the code that's alright so but when I pulled up player what I found is a lot of functions that do exactly what I wanted start play stop pause things like that I was like okay this looks good player is what I want right all right step four let the compiler help you write when you're compiling code when you're writing JavaScript I assume a lot of people in this room write JavaScript when you write JavaScript and you're not using compiler or something like typescript you kind of don't know if your code is going to work until you run it but when you write C++ code the well you also don't necessarily know if it's gonna work until you run it but there's at least the compiler that's gonna help you out along the way it's gonna say okay well you did this wrong for one you know so let the compiler help so the very first thing I did actually was I just took the compiler and I said try and compile player dot CPP I didn't think that this was gonna work but I knew the compiler was gonna tell me why it didn't work right so it tells me something right away lib mm XML file not found or the the header file that's what dot H means there so it gives me pretty easy error to tell and actually I remember because I built normally mmm XML that was one of the libraries that I needed so at this point you can kind of decide okay well am I going to try and compile this library I'm gonna try and cut it out you could go either way what I decided to do was actually just pull down the library I thought okay this library it sounds like XML it's probably not using any OS features I can probably compile it myself so I pulled down the the repo just like the other one I ran am configure that's another in scripting tool you just run that and it does the configure step for you and then I run make that's all I did and it just worked cool right so it generates this library for me that's what the dot a is stands for archive lib mm MX ml so that was exactly what I needed actually so then I just go back and these are sort of like magical commands that seemed they were not really that magical for C++ but like you may not have seen these before you do - I that gives you the include directory you do - L that gives you the library directory and then you pass in lower-case - L and that's the library you want a link and I did that and then I get a new error and so then you sort of go through this process right you see which errors you get and then you go okay well how do I fix this error so step 5 tip 5 do the simplest thing first it's so easy to just like dive in right when you're writing code you just go okay well I'm just gonna like what am I gonna do I'm just gonna start working on these things I always try and do the sort of minimal amount so in this case we saw before that there was this fftw that was the next header so I was trying to decide okay do I try and compile it like I did with mxml or do I try something else so when I looked at the repo for Paul stretch I found okay it actually supports two different FFT libraries so the f50 is kind of what you saw before that's a it's a mathematical function math majors in the room here probably did a lot of effort tee's by probably not by hand but you probably remember Fourier transforms so there's fftw is great library super fast super complicated there's also this kiss FFT kiss stands for keep it simple stupid I was like hmm which one do I do simple simple simple small it's already included in the repo so I did that and these commands look kind of magical as well but actually they were all there for me in the compiled script for me when I compiled normally so I actually just copied these and put them in so you'll see a few things that are a little bit weird there's this - X C here that says compile this file in C mode rather than the C++ mode and then acts C++ means go back to C++ mode that's all it is and then you can see I do this - oh just like Thomas did in his talk and then I get past it well look at that making progress so now I get to this audio file dot H not found that was the one we saw before right so along the lines of keeping it simple I looked at this header file and I saw some things here so what I found was it includes all these different audio file formats that's what the a input is that's what the Borba cinput is that's what the mp3 input is rather than actually porting these I actually thought well you know what let's do something even simpler these are all actually an interface they all follow the same interface I can just implement my own dummy interface so what I did was I made my own dummy input in the header file and then in the C++ file I found the exact same code see where it's creating Vorbis mp3 audio and I just said don't do that I do this pound if that's in C in C++ you can basically say if I'm compiling forums krypton then do this otherwise do something else so I create my own dummy input simple very very simple change like this is the first change that I actually made to the code and then you can see this interface input s this is the way you would do like a C++ style interface I just create my own dummy interface and you can see this is like all the code that I did return false return 0 and then for clothes & Seek do nothing right all that I'm doing here is saying do the simplest thing do nothing I just want to try and get it to compile and tada it compiles awesome right so I'm ready to ready to roll I got my program working well not exactly actually the problem is is that as pharmacist was showing webassembly kind of needs you to sort of tell you well what what do you want to be able to do with this code right so I have my webassembly module but it can't export any it doesn't export anything to JavaScript yet so actually I have basically a useless module but what I've done is I've gotten it so that my C++ code actually compiles and so now I feel more confident I'm like okay let's take this piece by piece so here's my tip that's a little bit different than what Thomas talked about and that's using this library called in bind so what's really cool about in bind it's built into in scripting so it allows you to basically bind JavaScript C C++ in a much much much more natural way you can take C++ code C++ classes functions everything like that and bind it directly and have it create the exact same JavaScript code so this is the imbibed documentation you can find it on M scripting org as well there's all the links for these things I'll have at the end so you don't need to try and guess what they're where they're at and this is sort of like my sort of intro to the to the binding right so what I did was I just write the the bare minimum here you can see that I've got this player class that was the same player class we saw before and then I have this start play function that's the same start play we had before and stop and pause and you can see there's there's not much going on here really all I'm saying is I've got this name that I want to sign too to JavaScript and here's the C++ code to go along with it that's really though that's happening here now by the end I had a lot more classes than this but this is kind of where I started and then you can take this and run - - bind that says use and bind for me and then I pass in some of the other cpp files what I found was that as I started to compile the code it would give me errors it would say oh well this this doesn't work this doesn't work and so I started including more CPP files the same ones that the native code was compiling and eventually it compiles and now I actually have something right the thing that I'm missing though is that I still have my dummy implementation of the audio so this point I am probably the most excited about Web API is our powerful this this is the thing that's so cool to me is that there's a Web API called decode audio data it's built in all the browsers that you use have it right and what's amazing about this is that I don't have to compile mp3 I don't have to compile OGG for wasum I don't have to compile any of these audio file formats they're all built into your browser right so I thought to myself okay well rather than compiling these into the web assembly module why not just leverage the fact that the browser already has it so what I did was rather than taking the the mp3 files and compiling those I created a little wrapper a little JavaScript input wrapper class and that's what I'm doing here so you can kind of see the same sort of structure as what we saw before there's this open closed read and seek and then you can see I'm mapping the strings as well you know the format is like okay maybe you have to sort of try and figure it out but the docs all tell you how this works basically all you're saying is when I call this C++ function call this JavaScript function just passing it straight through this one right here is a little bit more special right so when you're doing a read what you're actually saying is take this so if you if you've never if you don't know C++ this might be a little bit confusing but basically what's happening is you're saying I have a certain number of samples audio samples and I have a data pointer that points to that range of audio samples and what I want to do is convert that into something that the JavaScript engine can actually use because it can't use pointers from C++ so what I'm doing here is I'm saying well take that range of memory and actually could invert it to a typed array so that's all that's happening here the times two is also a little bit tricky but that's all all that's doing is it's saying actually there's two channels of audio so you need to have a twice as long buffer that's all that's doing and then in the binding layer I do basically the same sort of wrapping as before but now I'm wrapping it for input s so input s is are a pure virtual abstract base class and so I just map all those function names straight through and again read is a little bit special because I want to create my own interface for it but otherwise for a lot of functions you can pass them straight through and then finally in JavaScript code I can actually derive from this base class and create my own JavaScript base class and now what's really great about this is that any time the wasm module the web assembly module asks for audio it'll actually come out to JavaScript and I can put anything I want in there I could put a sine wave that I generated programmatically or I could put audio that I had decoded from decode audio data and so just to give you sort of a high-level idea of like what's happening there essentially I have this read function and it's got a pointer to that data it just wraps it it just exposes that and wraps it in into JavaScript and gives me back a you in sixteen array now this is something you maybe you haven't seen a lot of if you if you write even if you write JavaScript because it's a little specialized but essentially all is it's saying is that it's a very specific type of array its array that only has you int 16 values which are values from 0 to 65 files and 560 5535 so that's all that that's happening here that's all that I had to do and actually when I did that it was done like my code was like it was I was able to generate audio and to show it to you I actually have some fun little demos so let's see if I can do if I know how use a Mac alright yeah what is it command + + there we go yes alright so I have some demos here and I'm gonna start just playing this one [Music] all right okay obviously not slowed down demo fail number one let's see if I reset it if it works properly there we go so the thing that I think is so cool about this is that I didn't actually have to extract the audio at all this is an mp4 file and when I pass the mp4 file in to decode audio data it says great here's the audio track and then I hook it up to the video tag element in the browser and I just say play at the same rate as the audio is playing and now I can overload over lay those two pieces and I get the ability to create a video that normally you would have to create using like audio or like using video editing tools this is just in the browser so I thought this was so cool thank you I thought this was so cool that I went through and I and I and I did some other fun demos too so if you'll indulge me like I said I'm kind of an old guy so these are [Applause] [Music] all right so for for the young people in the room that's a Peter Gabriel sledgehammer he's saying you can have a steam train that's why the steam train is alright so I thought this one just sounded really really awesome four people don't know this one this is a meme called mr. trololo low I think and I just want to hold for this one note because I think it sounds so great [Music] I love that okay and so then last mo just something like kind of silly and fun [Music] [Music] all right so I apologize to everyone who's going to have that song stuck in their head for the rest of the for the rest of the night I will actually cut it off here thank you and actually that's all that's all I have so if you want to know more about all a bunch of this stuff here's a bunch of links I don't tell you what they are so you can just check them out and see if they're cool it's not then you know I'm sorry thank you [Music]

Channel: SFHTML5

Views: 3,706

Rating: 4.9607844 out of 5

Keywords:

Id: FQJrcX4Ae8A

Length: 25min 24sec (1524 seconds)

Published: Fri Sep 28 2018