Oculus Connect 2: Live Coding Session with John Carmack

I okay so we got an hour to talk about VR script what what led us here where I think it's going and all that so if I can get the all right there's my phone up here a lot of latency in that loop so the VR script apk right now it starts up and it's a it's an interpreter from script files and they're really simple this is kind of the home screen for some of the example code that I've written to just kind of point out a couple of the things that we've got where this is a picture taken with a theta with just a few text fields put up on top of it the initial application that I used as one of the straightforward simple things that people would want to do is basically just linking together multiple photos with voice and text annotations so there's things like okay let's put a an audio cue you should have audio coming from this okay so just being able to play I you know play audio there do some basic I kind of pop-up things and then maneuver between multiple panoramas you know so just like a few shots inside our office and this is just a really simple thing that I would imagine a consumer possibly doing where you take pictures with I you know with the 360 device and then you want to be able to string them together because the weakness that we've got right now is you can side load content you can kind of throw it into the 360 photos directory and use our existing viewer for it but all that does is throw them into one unsorted directory and you page through them it's not you know it's not a really great user experience the other thing is that that we're always coming up for me when I talked about the media usage is we support fully panoramic videos but there's a whole lot of other media systems or media formats like cylindrical panoramas and partial a partial panoramas things that are just wide-angle cameras and we have no really good way to to use those in oculus there in any of our platforms we our options are you can write a custom viewer for something where we've got a template program for a template project in the SDK that you can instantiate give it a name it creates this whole project but it's a whole directory full of stuff and it's not too bad to work with there but you have to import it into your workspace and start working with it it's just not a a friction free environment and it still takes you know 30 seconds or so to iterate for each cycle when you're trying to download things to the phone.i so some of the other tasks that I really wanted to resolve was this ability to make media formats so that somebody could have a new camera or something that would then be able to take whatever field of view undistorted generate the right mesh and show it in virtual reality and right now that's the process of either making a native project or I or building a unity project and something that I saw last year at one of the conference's we had a whole bunch of media people talking about their work and it's almost a funny thing about all these content people learning unity to be able to do these basic media things we would have the Felix and Paul systems where it's they're using unity to draw a panorama and a prop model it's basically to draw you know some one single model sometimes to cover up the camera spot to do a few other things it'd be nice to be able to fade programmatically all of these little simple things that game engines are incredibly overkill for and building fully native projects is you know a lot a lot more burdensome than it really needs to be so one of the positions that I was that I was at - and maybe early part of the year maybe in January February was I was thinking about this notion of experience files where if we define something where you've got a video stream and then a model file you could do a lot with that where you could make the model file the screen is it under sorts it properly it sets it up for that so that might be some simple declarative form of data where you bundle this together probably in a zip file where you've got an mp4 file with the video stream and then one of our OVR scenes for the model you could set that up and i and then play that back and it would automatically do what you wanted for that particular camera but it's a slippery slope that you get there where it's not long before you wind up saying I am okay well I want some little decision-making process here I'd like to let people choose one path or the other and we're starting off with a JSON file there like maybe we can put a little bit of logic in here somehow we can have a basic if statement and then we need some expression Parsi we want to be able to add some things to that and it's a very slippery slope to writing a very bad programming language so I had decided that I didn't want to do something that was really ad hoc like that at a minimum I was thinking I could put in kind of the most simple language that was fully featured I that I might want to embed sort of inside a JSON file and I was thinking scheme at that point like something like one of the tiny schemes or something really really small and simple non performant but it's at least a real language that people have been writing code in for for many many years and I was leaving I had left it at that for a little while working on other projects and then later when I started working on the social projects I did some of the basic VR UI for selecting avatars for selecting rooms to go into drawing the avatar heads and this was all in raw C++ and it was fairly miserable I mean you you just you know you got your way through it but it's not a really pleasant programming experience there's just all of the problems of working with C++ the wrong abstractions and so that's certainly what we have set up as our API is in in C++ but it reached that point where I wanted to do more things for the social I wanted to make more things we want board games we want card games we want these simple things that do not need game engines fully but I wanted to be able to develop them fairly quickly and let other people develop them fairly quickly and that was when I started making down this path of well let's consider making a full-fledged interpreted language that we could write VR not applications so much but very simple things more like web page level content the type of things that we're done early on before it blew up to sort of the stage it is today and because I had been using racket which is a scheme Lisp for some of the social work I did the server's originally in that I looked at and I thought well what I really want from a development system is to be able to make this really fast iteration times I'd like to remote develop onto the phone so if we can switch over to the PC here screen alright so this is the the dr. racket IDE for real old timers it kind of feels about like turbo Pascal or something where I which was the good old days really in programming in many ways so to write the kind of simplest possible sort of hello world in what I've got here this is running on on a Mac and then it talks to over TCP to the the phone here running the the VR script apk so the initial thing part of racket is you define what the the language that you're using each and the initial thing in so I it starts off defaulting to racket but racket base is a smaller subset from it the VR script part of it is in I the VR racket I am module setup and then the connection that sets things up initially we define some kind of a frame function that will initially just do the simplest thing put up on the hood hello world and then we talk to it by calling VR main and I go and I get the IP address of the phone which is kind of shown on the home screen here and what the initial things gonna be so when I run that if we can switch back over to the phone so there it's kind of hello world running on gear VR as a new program and that would be the HUD thing makes it face lock but there's lots of other text things that you can do the setup but the idea being there that that's that super super quick one file access now if we want to start putting some other media into it I've got everything is set up so it's pulling it's pulling media from the internet and caching it so if I wanted to replace the simple text to add a background I can define a reference to a URI you can use these inline anywhere you want to but defining them ahead of time lets it be pre cached and then I can go ahead and make a model instantiate that in their model and we'll put it at the origin oh yeah I'm sorry screen back so there's gonna be a lot of screen switching so back to the phone it'd be nice if I could have gotten two screens so here's a model that was loaded in from our OVR scene format then thrown out onto the phone now this is something I'd actually be I would love to hear suggestions for a better generic model format this is something that for years I have not had great closure on where for for texture formats we've got lots of good very native hardware like formats that are easy to load things up for models I did another search last year for oculus starting this up and I we still wound up kind of with our own bundle format and we have a tool when offline tool to convert from FB ex's to our OVR scenes this is how our theatres and our prop models and things are made but it's not great and I wish I did have a better solution but so this is a so we're in a model scene that's positioned with the origin kind of right at floor level now which is not what I want so go back to the the desktop so instead of putting it at the origin let's translate this is opengl coordinate system looking down negative Z Y is up and X is to the right so we'll translate it up to meters oops wrong way negative 2 meters okay back to the phone okay so but that's the type of kind of feedback loop that I wanted so much that is missing terribly from gear VR development where there's tons of times when you say oh I meant negative - and Yuri compiled it and it takes 30 seconds maybe for your native stuff or if you if it was in unity code it might take a couple of minutes to rebuild the project and push it over so I that was my kind of core goal of what I wanted from the development side is to be at this I am this fast interaction and being able to quickly change things and be able to make it so there's not a whole lot of boilerplate one of the I think early success stories from this is I when we were doing a demo at Samsung relatively recently we had heard that the I the executive we were demoing - was begin to travel and all this and somebody said well let's let's make a travel application let's take a bunch of panoramic photos and add some voiceovers and add some pop up and pop up the images and various things and I set up some basic basic UI just here's a button that will pop up a picture and here's how you swipe to go to the next ones so I wrote several hundred lines of code there and then Matt Hooper our creative director in Dallas basically went and sourced all the images put them all in typed him in and and got it going and that was that worked really remarkably well so the conventional wisdom is that a lisp can't succeed that I am you know that this type of it's got parentheses instead of in fix notation prefix in parentheses instead of in fix that that's too alien that you know that programmers won't deal with this I mean you could say that the evidence supports that assertion based on program programming language adoption but I you know my 11 year old son is programming in racket or a creative directors programming and racket it doesn't seem that ridiculously complicated for me and there's a lot of interesting things as a programmer right I'd kind of recommend learning a little bit about Lisp it's got a long and storied history and there's some valuable lessons to be learned from its time but on the other hand we are already working on setting up a system like this so that we can use JavaScript or potentially other languages so architectural II what's happening here is the the VR script apk acts like an ex server where it's communicated with only over a TCP pipe when I'm running it on the desktop this VR main function actually establishes a connection talks to it there it gets a it gets an input struct set over to it and it writes back a frame struct with everything that's going on when you're done with developing it then you can put the the script file either on the internet somewhere to be downloaded or you can put it on the local you can push it to the like SD card oculus VR scripts and then that home program that lists my few samples you look over to the right and anything that you've put in there shows up at that point so you can write tiny little applications here and then very quickly kind of put them onto your phone and then run around with the phone without the PC showing demos there there are big open questions about exactly how we want to deploy this the first thing that will be made available to people hopefully as soon as the next couple of weeks depending on how my schedule goes with some other things is the ability to use this as kind of an apk development tool so that you can create something if you're just doing a simple media application build it simply like this and then package it up into an apk for submission to the oculus store for going through the conventional path we are having a lot of discussions internally relating to security and various other aspects of this but my hope is that we're allowed to do this in more of an open way where it's more like you can pass a link around and then somebody can click on it in a webpage because that works pretty marvelously where I register the VR script scheme on this so when this apk is installed any URL in the system that's that handles it properly if it sees a VR script colon slash slash and then a net address it launches VR script if you're browsing in the network browser kind of one of normal browsers for the phone you click on one of those links it says insert it into the hmd you put it in there and then it runs that script so let's go back to the code to the PC so let's I get some other things working in here pull in some other media assets so the idea was I was going to start something that would be like the there's a Kirby's ninja dojo on Tendo that I play with my kids sometimes throwing throwing stars that target that I you know inside a scene here so we've got a few more pieces of media that I can pull into this lets get so we'll take a target copy that now one of the other things that I would say about list is I can imagine that editing lisp in notepad could be fairly considered user hostile I am but if you're in an IDE that does parenthesis matching and auto indentation and all these other things it's really really easy so let's we'll push this to like a negative two on Z all right back to the phone alright so there's a target and back to the PC and head back to the phone and there's our throwing star now let's make some things move back to the PC all right so let's define a function for the start or to the target position and there are the harness for this code that's running here is all defined in the the VR racket file that kind of wraps the mainframe function so it it basically parses some input that comes from from the phone where you've got touch joypad different pen but you've got all the basic VR information that we have for our native applications as well as time and gaze and you get that all pulled out into variables that you can access so let's set this up to be we're gonna make BEC 3 we'll have it move in X just on the sign of script ii 0 y negative 1z ya notice that the parentheses matching just is very very easy to do there and we'll change the star model to be translating again working in an IDE is nice rather than passing it all the command line oops I'm moving the wrong way so he can switch back to the end of the phone so I was moving the star wrong star model what did i do there and that actually is one thing I do intend to put a console in VR so I get asked that all the time about when do you get to program in virtual reality and I think that's in general a terrible idea I am but I everybody asks about it so I you know we will make that happen sooner or later but at least we're getting proper error messages do you want to do it all right so there's a moving target now the Wi-Fi here is not so great so it's hitching around on frames so this is frame synchronous where I am you know it fires I it fires one frame of input over the TCP connection that gets the frame back so it does all run with asynchronous time warp so it's smooth on your head no matter what the frame updates are happening and in fact it's a little bit better than the basic like the way if it was locked in per frame in the native code and you moved your head too fast and it didn't get an update you would have the black pull in what happens with this where it is decoupled enough so that it's running the VR frame loop potentially multiple times for each script frame loop so if you stop updating the script the animation will be hichy but you can look all the way around behind you or whatever so let's play a sound let's get some more of this data from here so again these library level functions are in the VR racket where we've got precedent you know all the different actions that you can do on the touchpad with swipes and buttons on the joypad okay so we've got sound there so now I actually throw the throwing star we'll have to define some state I'll do this a parametrically so I'll set up a start time for when you when you throw it and we'll calculate a trajectory just based on the Delta from the x one of the coding conventions in lisps usually is that global variable is if you choose to use them I you put stars around them or earmuffs as they're sometimes called [Music] so one aspect about the programming language is in general where if you had asked me a year or two ago I was generally pretty down on multi-paradigm languages like Lisp where I have a high opinion of functional purity and I started out writing this in sort of a pure functional manner where I like especially if you look at some of the racket tutorials and the realm of racket you'll you'll write games in this purely functional manner and it's you can certainly get that done but it was awkward enough that something that is has an uphill battle for adoption in general i adding an extra level of awkwardness i thought was ill-advised so I did kind of embrace the global variables for single file scripts but one of the nice things about racket specifically is that it's I you know it's it's a 20 year old language development system and it is extremely powerful in a lot of ways I'm not really a language advocate you can do anything in any particular language so I'm not going to try and say it's the best language for tasks XY and Z but it has it has statically typed options with typed racket you can do custom languages to change all sorts of different things and it's got a wonderful module system you know we should only have something so nice and C++ in general so it's not something that while I'm setting it up right now these are all single file scripts it's not something that if you happen to write a hundred thousand lines of code in it you would be up against the limitations of the language so I think there's some really positive aspects and in fact that's a good time to mention that when I started this out up until last week I was using two different scheme implementations so I was using racket for the PC side of things and then I was using a smaller implication implementation called chibi scheme as the embedded part and my my rationale for that was there was only it's 5000 something lines of code where I could go in and I could fully audit that understand it myself if necessary while racket is a 20 year old project with alarm you know a much larger amount of code in it and I can't say that I deeply understand any of the inner workings about it but one of the things that writing documentation focuses you on things that are awkward about it where there's a number of differences between the minimalistic chibi scheme which is a revision 7 basic scheme I you know you could do anything you need to in that but racket is much more of a bells and whistles kind of descendant of that and there's a lot more that you can get done in there and I was finding myself over and over doing something in the remote development and then when I deployed it onto the phone it no longer worked because something was a little bit different or that particular thing wasn't there so last week pretty much while everybody else was practicing their presentations for the the keynotes and whatnot I was swapping out the language infrastructure in VR script so we're lucky if nothing catastrophic ly blows up today but the bottom line is that it's the full racket running natively on the Android system and that's also the architecture plumbing that that's going to allow us to switch to a JavaScript back-end as well so we've got a start time I will have a throw position man let's define so the position will be let's make a start so we'll start it like point two meters to the right so the unquestionably Lisp set up like this winds up being a little bit more verbose for things that would be that you'd use operator overloading and C++ for so like vector things here I've got AB 3 and scale 3 and that could be set up as intra as kind of built-in operators and you could wind up doing that's one thing about racket because you've got full control of the language you can do anything even writing in fix operations I choose not to do any of that because initially I was portable across schemes and I'm still unsure about the value proposition of radically changing the language definitions directly so it doesn't bother me too much to have you know little bits of extra typing that wind up being necessary for this so we're going to add the star start position and then the direction scaled by the Delta in time from when we started it so when you have the pressed action we are going to set these variables and as always useful when you're working on something new having a bunch of the samples open in other tabs there I'm going to go look up what I have for the the view position so the the pose that had tracked pose is passed in as one of the input parameters and it automatically builds an inverse and because that's commonly used and then we'll also need to know when it's actually hit something so that it can stop moving oh and you have a debugger works in here I it's not the greatest debugger but you can step through but you do have little printf Oh Wi-Fi isn't very bad here now I am I have a fallback position where you can use ADB I am to talk to it if Wi-Fi is really not working well so one of the nice things about using this is it actually doesn't require the Android development tools to be installed so you don't have to go through the whole mess of doing all that so you can just have an apk installed and then using your desktop PC it's convenient sometimes to to have it so you look at logcat or you can use adb pushes and things like that all right so let's calculate our star pause and use that now for the drawing okay so the star is sideways so we're gonna rotate that now so we'll do a compose one of the kind of interesting things about lisps in general is the identifiers syntax is much broader than most languages where here in this case PI over two is an actual identifier and a constant rather than a an expression that's one of the little side benefits you get from the more uniform syntax okay so there's our throwing star now we need to do some hip detection so when we hit the target we want to stick in the target so we'll define an offset and then find a we'll do it when and then we'll play our sound there that should have made a thorough sound I'm gonna switch over to a DB here it works it's an ADB forward and then use okay there we go back to the phone now going over ATG but now back to the code I have an error so this is why dynamic languages are in many cases inferior to statically typed languages and again so in general I am very much more fond of the dynamically typed languages let's go much faster let's change the speed so the star position let's scale the star Durer here okay okay so well turns out it looks like this live programming is kind of boring so I'm I'm being everybody being quiet while I'm going through this but this shows basically what I wanted to demonstrate about the the quick iteration cycles and the straightforward ability to use this let me try this again will be probably a little bit of a boring segment here for a second but let me try putting this up on the web and then building a link to it so this is bio c2 racket so we have just an s3 I console here let me go up to some try putting this in the scripts directory okay so that file should now be publicly on the internet and I should be able to make the post from here you try making a quick post to the oculus developer board oh I don't really want a phone type this long URL well let's just pretend that works mm-hmm but the idea being that that's up there then and then the link if it's visible in like on a board on the forum you could click on that and then it would say inserted into the hgmd download anything necessary and then start executing it I am so that's not we're still we have a lot of discussions about security and sandboxing and implications about letting people download arbitrary script code where it's not native code but everything has bugs there's I'm sure there's some way to break out of the basic performance sandbox from racket I you certainly have to break out their form break their foreign function interface and all sorts of other things so there's a lot of work to be done before we could let that go out into the wild but that's an interesting model when you start seeing that that's what I would desperately desperately like to see in a development community of our on the forums people are saying well here's what I was trying to do here's my program here's the code with one link and then here's the executable link with the other one tap on that see what it does and get that kind of really quick back and forth communications and iteration so I am open it up to questions answer any questions about what got here if there's something that you'd like to see how it should be implemented in this be happy to talk about it so we've got I think microphone right here hmm hi I really like the the really fast iteration that you just like you were able to do a lot that yeah I think we'll really blew me away was mostly the fact that yeah you don't really have to set up a whole development environment which is something that's like a whoa when you're switching computers every time then you're trying to just like develop something aside from that at least I'm pretty sure most of us here are pretty much in tune with the native side I don't think unless we've had any particular issues that like switching computers or anything at least in my opinion I don't really think I would need to switch aside from that so I'm guessing you have a particular target audiences for people who are trying to just get into the space I mean I know the problem you're trying to solve is mostly rapid iteration prototyping and stuff so where do you draw the line between going from scripting to just native I mean what who is so well first I would say I bet you've Mis estimated things would everybody raised their hand that's built a native SDK project about a quarter so that's actually pretty good because our I there's a little more than I would expect probably because it drew a more technical crowd here but in general I and not that many people are like 95% of the applications are unity based there's not a whole lot that I are done native native does let you get the I am the higher performance features like one of the things one of my goals for this was everybody's seen my critiques of a lot of of other applications and I'm always pointing out things about the rendering and aliasing and so on by default this does everything right it runs in gamma correct srgb space it has MSA on the fonts are all signed distance filled fonts when you throw up a picture it winds up feathering the outline and it automatically transitions between the I'm I the overlay planes to use that for much higher resolution in fact if we could go back to the phone for a second the show we show one of the other scripts as I am as the example of that I am because I've been pushing for a long time to try to get people to use some of these features more significantly but like in this I in the VR tour application this uses when you say I want a panoramic background it's just the one line command and we can turn the volume down on that again so if you throw in one of the the 1536 cube maps it automatically uses that at the full kind of high peak resolution which I the people that just kind of standardly do things in the game engines wind up at the much lower resolution and then one of the things that was I was very happy about doing on this is we've got the focus points here so right now it's using the overlay plane the time-warp layer for the background but when you pop up an inlay picture it then switches to using that for the peak resolution so you can't tell here but in the head-mounted display that has over twice as many pixels as if you just kind of draw something in the standard way so getting the best quality which I nobody has done these things yet I wish people had more on just media viewing applications but I wanted to make sure that these best practices happened pretty much by default I but then yes that the hardcore programmer are serious native programmer that's building a big application I would not expect to use this there was a debate about or a discussion of should we port cinema over to VR script no of course not that's I that's a big application that has a lot of code in it already there wouldn't be a real benefit for moving it over this is mostly for developing kind of web-scale experiences where there there are so many things that on the web you would not pay 99 cents for but the derived value is in the aggregate where you could look at a huge number of things and they sum up to a lot and I think that there's a lot of cases of content that somebody would not go and create it and bundle it into an app and submit it to the store but that's gonna be critical for the long-term success of virtual reality where people taking snapshots with lots of crazy different cameras how do you get that into gear VR right now if we had a ton of media applications and people had demonstrated all that and it just it just worked we had our universe a media player that did the peak quality work then I wouldn't feel you know as much need to to work on this but if we had I am there is value to doing experiments with this I use this now on a fairly regular basis so just last week they I got access to some of these lunar panoramas that were in a cylindrical panorama projection format that again we don't support in 360 photos I could have made a template project started up and done that and that would have taken you know maybe a half hour to get everything right instead it was just you know command T open up a new tab in racket and write out the five lines of boilerplate then start building that and it was in that one file kind of application and that's something that I do miss from the big projects it is it feels ridiculous in many cases to create a solution or a workspace and then a project inside it and then pull over the templates stuff to build simple things this is in no way a replacement for highly sophisticated apps if I you know in the best-case world we see I mean people would have said that about JavaScript a long time ago as well and things grow beyond their original intentions sometimes but I would say I have my clear bright lines where if you are developing a skeletal animation system or if you need a skeletal animation system a particle system or a physics system you should clearly be using unity or unreal you should be using a real game engine if you have to do a lot of interaction with Java or other native code or something then you should be writing a native application but I think that there's going to be a large number of these simple things because the other driver for me on this aside from the media was social kernels that was one of the things that we we really found we're just putting the avatars and having the chat rooms if there's nothing there just the social chat room you say gee this is neat I'm talking to this person but there's not that there's not a draw to stay there but almost any little thing can provide that kernel so I think that there's a lot of opportunity for doing I guess I can show one of the other sample applications that I did for this [Music] so the idea of doing I really trivial little things but okay you got you know got an environment you've got some commands and then you've got a game board that you can sit down and weigh I expect that will wind up having AI you know all the basics of board games and card games but they're still clearly even just right inside oculus we have things like games that are being developed with unity we have our native applications like video and 360 photos that are being developed natively and then we have some social experiences that are being developed with this so my sort of core thesis for this with that there was a necessary middle layer or a useful middle layer between them awesome that's perfect action Thanks hi John yeah thanks for the talk and it's I think my favorite feature of VR script right now is the the speed of development I mean we saw you make changes and instantly you just see it right there fumbling around up here making it absolutely also not rehearsed like that's okay I guess my question is a couple things actually is VR script limited to gear VR right now is there a PC version that you can maybe create VR apps on the PC and is there a way for submission to the App Store that we can package them in a self-contained app without having yeah have an apk and then download scripts so the plan right now is Michael Antonov is working with me on both PC support and JavaScript and we're these are all no firm strategic decisions on any of this have been made but the go-ahead that I have right now is to allow this to be used as an apk creation tool some of that code the URI is the saw me typing in can be done they support regular HTTP connections they also support file schemes so you can do this without a without an internet connection at all if you don't mind ADB pushing the files onto your phone which in many cases is is more convenient than pushing them up to a web server and then there's also the apk : schema that I've got for that so you can bundle all the things into the apk and then I and then use like that so sometime hopefully in the next couple weeks there will be some preliminary SDK kit that is basically that lets you do this type of work I I'd be surprised if somebody puts something together in you know in the next few months that they wanted to submit to the store but that option would at least be available I think that we have to go through a more you know kind of a learning curve time where it's pretty raw right now in the future sets and what I'm hoping for is we get to this forum situation where people are trying things telling me what's obviously missing because right now it's developed around my needs the things that the tasks that I'm working on I'm sure there's enormous blind spots in it thank you I have two questions one is is it possible to extend it for example let's say some native code function so I don't have a firm I haven't formed a good opinion on that yet I am there's a lot of that opens up a whole lot of issues there I think that I'm going to be able to I'll certainly like to a chroot sandbox and probably a you know hopefully carve off all the system calls so it might be possible to do I am just like number crunching type stuff without being much of a security problem but I think I would lean towards saying if it's really performance intensive to just build a native application it may evolve that way but I think there's clear enough lines where I don't want to go down that too soon it does but be obviously in the privileged position of I add a new message type for anything that that I want to do myself but I so you mentioned there's the TCP pipe do you like swap out the front end for another language yes completely and that was I my client and all the first thing that he did was basically write a JavaScript using a JavaScript program using node that emits s expressions for this so the wire protocol is s expressions it's got the list para de Jan I do think that's actually one of the strongest points about list with the s expressions and the readers and the writers but you can emulate that in any language you could write a C++ front-end for if you wanted to it would just be a you know a poor choice for that but the JavaScript implementation talks to it it just the apk has no idea what it's talking to it's just got a protocol and I think that's that's one of those things where the it's easy to rebel against that as a performance focus to engineer it's like what we're Martin we're putting everything out to text in between but these are we have a lot of performance Headroom in that type of processing where thousands of times faster than when we might consider something like that I as I you know a funny point I did consider all the way back in quake one using essentially text strings for all of my media identifiers and that was you know like a thousand times less powerful computer I wound up not doing that but it was a near thing and we ran like that for a while so I don't feel too bad about doing all that it probably does cost us you know a millisecond or two doing all of the transport there where if it was all native and that was one of my reasonings for wanting to use TB scheme embedded is I had all this possibility I could rewrite the garbage collector I could make MEK threes and Mac fours native primitives that you know weren't even boxed i there was lots of things that as the optimizer you know the programming geek I would be thinking this would be fun to go extend all of that but then I just realized that's not a good use of my time and just making it it's a completely forked process now which is really kind of lovely it's an unmodified racket Android system there and whatever other languages we put there they're just gonna be four processes with pipes set up for standard out standard error and then they connect locally over TCP so I'm pretty happy with that there's a performance cost to it but it's extremely flexible thank you thanks for the talk just a quick question here maybe just shot in the dark is there any plans to release some sort of lightweight IDE based around VR script well one of my main reasons for liking racket is that the dr. racket IDE is you know works on PC Mac and Linux and it's a single click download it's it's really great it's not Visual Studio I in terms of all the feature sets but for making these relatively simple things it's very close to what I would consider ideal for for building these small things so I thought that was one of the big Wynn's thank you so one of the neat things with this is that it's very real-time you know you can add changes in and see them pretty much instantly but it seems like it's also lightweight enough that in theory you could do the coding in the headset as well be able to type it all out and I've seen people do that before is there any plans to maybe add in some kind of ability so that you can type your code with a keyboard while in the headset and then see the changes live so I I say to myself clearly that this is a bad direction but it will probably happen accidentally anyways you know we're going to we are gonna get an error console that's so like all the standard out stuff all your printouts go to something that you can pop up while you're in VR and then it's just such a short step there too it's like okay allow somebody to have the Bluetooth keyboard and type some things in I and use it to evaluate expressions or communicate some way so I don't believe there's value there but I believe it will happen sooner or later just because I there's this pole to do it I mean it's so commonly heard as a web developer this kind of thing really appeals to me I was I was wondering if you've interacted with the Moz VR guys or guys so when dr is i so it's worth contrasting the the layers of abstraction here we're web vr is a thin wrapper on top of like the oculus sdk that lets you use WebGL to draw the eye views importantly what I've got here is it's not a retained mode seed graph where it's where it's retaining information state in the apk side of things but it is surface based where you specify models or surfaces and you're not drawing each eye that's important for me because we want to be moving everything towards using the multi-view extension which we're working with arm and qualcomm samsung to get adopted into these because it's going to be about a 30% cpu improvement for us on the phones where we specify everything one time and then it can replay it on the GPU for each eye and it also allows us to do environment captures from you know from a single scene description so I the Webb VR is is not something that I think has well I got to admit WebGL has has succeeded far beyond what I expected it to so I don't write off web VR especially on the PC but it's a harder situation on mobile where everything has to be much more perfect and the a lot of the value that's coming from this is making everything use these optimal things like the time warp and the right rendering setup where if web VR starts running on on gear VR it will be sort of the lowest possible common denominator for how how your models and geometry and everything would show up there there may yet be value I'm not sure that crossing the gulf of performance between the PC where you can throw ridiculous horsepower at it's not like your web VR script on the PC is going to run on gear VR but if you built it the other way around then it likely could I you know I I wish them the best on all the Mozilla team working on that but I think that the being more directly tied to our strengths in the SDK was something like this is you know is a better bet for us being so early in the process it's kind of tempting to use Vulcan testing with the R script what are you thoughts on value added with such an engineering approach with VR script and then just letting Vulcan be as it is and see what happens so I have had surprisingly good results doing a lot of little OpenGL hacking with this where I'm just like let's set up a program and even just a matter of iterating on program you're iterating on shaders here where it is because the shaders are just text in there I can change something save it and reload as opposed to I've probably done hundreds of cycles of that with my native code with a 30 second development cycle where I'm like just change one thing and it still doesn't work so it's been kind of surprisingly valuable for hacking around on some basic graphics things I'm and we are trying to make this more block state based we're very much considering how we want to do this surface based approach with what what in Vulcan we're going to wind up having setup for it one thing that I know I have ahead of me is right now there's no direct issuing of drawing calls in here because I want it to be this eye surface based for the multi-view and everything but I know for drawing into FBOs and generating textures that I'm gonna have to eventually have some basic drawing interface and I'm not sure exactly what I want to do for that I am making it easy to test that's always been a problem for testing new hardware api's and things like that where if you've got a long iteration cycle of course but the Vulcan we're probably gonna have to worry about crashing the phones in the early days as the drivers mature and the development environments not going to help us too much they're there there might be some utility to making a whole set of bindings to be able to do that but it will be that's one of those tough value calls because there's gonna be a large amount of setup to get all of the all of the defines and functions and interfaces set up there do you save enough on iteration there I think my gut feeling is it would probably be a net loss but it might be an ear thing Thanks hey John so you mentioned I think three levels of destruction the engine itself racket and then JavaScript I was curious as to why you still my racket is the middleman of sorts no it's not it would be JavaScript would be replacing racket entirely so the APK has its ports open and you can you can basically fork off the racket interpreter or fork off whichever JavaScript interpreter we wind up going with and in theory we could have Python or whatever else you want there somebody would have to put up a you know you'd have to write a library that knows how to speak the wire protocol that I've got but we could potentially support a bunch of things the VR script scheme would point to our apk and then we could look at the file extension to decide what interpreter to use I mean it would be best if we don't have a broadly scattered language infrastructure I am you know we'll see how the general opinion is that if javascript is available everyone will use that but I think that some people might be surprised at how kind of nice and convenient and sort of fun working with racket and a ID and everything across it is just a shout-out to John Clements because he deal alright and we're only one and a half minutes overtime budget so that was okay great I joined I'll make this quick I was wondering if there any sort of security or performance concerns that you had in mind about downloading the our script things over the Internet so when I was watching it the first thing I thought was this would be really cool if this was kind of like a gear vr web browser and you could download the different racket files from different places are there any concerns about there are enormous concerns about all of that where of course in theory you have an interpreted language you can sandbox it but of course in practice that is not and has never really worked out with the bulletproof nature that we'd like to have we are you know we have some meetings scheduled to to bring in we've got a lot of experience at Facebook with this type of thing and it is outside my realm of experience left to myself you know I would sort of say hey it kind of works let's go see what happens and that would probably have been kind of bad i but we have experts you know we have domain experts on a lot of that at Facebook and we're starting to bring them into the conversation now so everybody's comfortable with releasing this as like an apk creator where it would go through the same approval process as any of our existing apps but I want to get to that more open space but it's just gonna take longer to make sure that you know and make sure that we're not messing up art you know we're not a vector for bad things to happen to our users and when do I get to play with this so depending on what I mean I've got this list of things that I get to figure out what I want to do when I get back you know get back home tonight the our script is one of my major initiatives and I you know it could be as soon as a week from now if I build the apk packaging tools and put something out it would be pre-alpha very rough we have a tiny bit of setup documentation that's been written for it I again personally I lean towards getting things out in a rougher shape I admire the Facebook move fast and break things mantra about it I would like to iterate to success rather than trying to build the perfect release but it depends on other things like if I you know if Mojang let's be back into the codebase I'm going to be working on minecraft as my top priority for things I am and there's a few other possibilities but I hope within a small number of weeks at most that something will be out there to play around with is I think even at the even at the level that it's at there are a number of media applications that I would like to see people write just if it gets to the web the web page sort of things with links my vision for this would be every content creator every panoramic photographer that has a web page or a web presence also has the little link that goes to their VR gallery that lets you view things I think that lightweight stuff has a lot of value and I think that this can be a really good enabler for him thank you hi John I'm Reed this is the last question so I just wanted to say this is gonna be easy things so for the last 20 years of my life you've been such an inspiration to me I've had some times of darkness and it's been amazing today is my birthday I would I would love to like go up there and and if someone could take a picture of me with you that's all all right four and a half minutes over budget good timing yeah all right [Applause]

Channel: Oculus

Views: 116,528

Rating: 4.9470434 out of 5

Keywords: John D. Carmack (Video Game Developer), Oculus VR, Session, Virtual Reality (Media Genre), Coding

Id: ydyztGZnbNs

Length: 65min 21sec (3921 seconds)

Published: Wed Nov 04 2015

Reddit Comments

[u/planetary_dust]

This has been a pretty long time coming. At about the 2 min mark he goes into his exploration of functional programming. He tested Lisp, Scheme, Haskell. While he felt Haskell is more suited to what he needs for game dev, he concludes that something like Scheme would be great for scripting an engine etc.

https://youtu.be/1PhArSujR_A

[u/cammil]

As a Lisp noob, what should I be seeing?!

[u/strranger101]

oh, this is beautiful. I've been burned before, but I'm ready for this to mean something. Any guesses why he went with Racket?

[u/bitwize]

I think Oculus actually settled on JavaScript as their VR scripting language. Because, you know, JavaScript is an acceptable Lisp.