What Does Programming Look Like In 10 Years?

what does programming look like in 10 years right and my first sort of real approach to this my first actual time and I was tasked with trying to figure this out of it was as Fernando said and Microsoft so I was a program manager on visual studio at Microsoft and as time went on I originally started as a program manager on the editor and eventually came to own the experience of using c-sharp and visual basic and while I was there I sort of carved out this little interesting position for myself which is trying to figure out what Visual Studio will look like years from now what is the future of Visual Studio if we were going to make this programming thing better if we were going to make programmers more efficient what would it look like and in order to answer that question I managed to convince my superiors to let me do a study and I and so I did the to my knowledge only ever done and to end study of Visual Studio where I sat behind a one-way mirror and watch people program and solve problems on a computer and this sort of very general set of tasks for a month and a half right and as a result of that I gained a lot of sort of insight into what it means to program what it means to be a good programmer right and how we actually do it and this is important because as an industry we're actually not very introspective you know I loved it we would have these guys who'd come in I only use the keyboard I never ever touch the mouse right well we have videotape of them using the mouse about half the time right it turns out that depending upon what tasks you're doing what what your mental processes are kind of doing at a time you have different modes of operating and one of those modes typically is done with a mouse not a keyboard and so I I did this this study and started to design what I thought the future of programming would kind of look like it's specifically tool oriented right like that that was what I was tasked with I wasn't looking so much at the languages or any of that there was somebody else whose job that was and so I built a bunch of interesting you know prototypes and stuff and they've been slowly trickling out of my or soft since then but this is where I first started right I started at Microsoft on Visual Studio imagining what programming will look like in 10 years and I'm like it would be really faster right right that's that's kind of where I got to and it will be it will not only would be really faster right but you'll be able to navigate really efficiently right these were the two things that I sort of latched on to and that study also taught me something very important and that that what that thing was led me to the next project which was lighttable right and what that study taught me was that despite as Fernanda said Visual Studio is a monument to humanity it is 55 million lines of code when I was there five years ago right it is a gigantic piece of software but in that study I found out that most of the things that are in Visual Studio weren't actually helpful they could have been right they could have been useful it wasn't that the features themselves weren't good but they were dwarfed by the fact that everyone was playing computer inside of their head this was how they came to understand the software that they were working with this is how they came to understand programs they were unfamiliar with even once they were familiar with when something goes wrong right your only action is read some code imagine what the computer would do right and play it out in your head write some stuff down and just keep doing this over and over and over again until eventually you have basically simulated the program and you found your error right the premise behind lighttable was what if we were directly connected right what if there was no time between change and result what if we shortened the feedback loop to effectively nothing and I think you know there were many interesting ideas I think in my table but the one that people really latched on to was what I called at the time the insta repple and the idea was you typed code on the left-hand side and if there was anything that ran as a result of that on the right-hand side you'd see the code filled in with all the values right so if we you know kind of zoom in on it right so you know this is enclosure so it's a lispy syntax but if I add 3 & 4 it shows me that the answer is 7 so I in do some math and then what's interesting is this this def end thing is a function right and so the function takes two arguments and you can see they got filled in as well and you can see what the result is and what this enabled was as I typed it did this so I could literally watch the data flow through my program right I there was no question what was going on when I said that people were forced to play computer inside of their head what they were trying to do was make the program observable right they were given this sort of opaque thing with a set of instructions explaining how it works and the goal here was let's just open the box and let you see inside of it and that was that was really interesting and it was useful and there's lots of very neat things that came out of light table I mean it had other ideas too you know it had this sort of small talking notion of like I'm gonna put code like little snippets of code all over the place and embed running applications in it so that you know I can sort of create my own organizations for code right I'm not limited to just what I can put in a linear file I can create this sort of canvas upon which all this stuff sits so we did light table and one of the things we we had sort of an explicit goal of right was not just unlike a Microsoft where the goal was purely like make programmers more efficient our goal with light table was actually yeah we want to make you know me more efficient that's great and all that has potentially huge economic value but more importantly we wanted to make programming more accessible right one of the things that we we came to was this idea that the reason programming is so hard is because it's unobservable because you can't see what's going on you can't see the effects of change you can't you know watch your data flow through and so you're doing this this replaying in your head and like table you know with features like this yeah any we're pretty good for programmers actually you know that that study that I did for Visual Studio we kind of replicated it with light table to see what the difference was right and for a career programmer hey programming this way it's about twice as fast and there are lots of interesting things that came out of that for example in Visual Studio people would write large swaths of code before ever trying you know before ever pressing the play button basically with this model people wrote little tiny bits right and we're trying it all the time and as a result of that they built up as they were building up sort of abstractions and and you know all these functions and and this sort of set of primitives they knew what every single one of them was doing they knew it worked correctly they knew what their result was going to be and so that was great you know career programmer could be maybe twice as fast turns out that twice as fast actually isn't enough you know this is this is something that people have sort of a religious attachment to right these are our tools these are the ways we express ourselves as programmers and so 2x while interesting and compelling is not enough to get me to drop Emacs right or is not enough to get me to get out of Visual Studio or whatever the case may be so it didn't really get the the sort of attention we had hoped for in that regard although you know we got tens of thousands of people using it more importantly though it did nothing for our other goal this goal of making programming more accessible and we try to understand this and that understanding resulted in an essay that I wrote a while back called toward a better programming and in it I sort of realized that I had approached this problem you know I I you know I used to say I want to make programming better right and I realized I had made a classic rookie mistake which to this room is probably well known and understood I try to answer a question I didn't really understand right like what is making programming better mean what is better that that implies that there's something wrong what is wrong with programming what even is programming right is kind of an interesting question to start to ask and so we started doing some research we asked a bunch of people build a bunch of prototypes tried to sort of develop an understanding for what we were actually trying to do and I sort of came to this to an answer about what is wrong with programming if we're going to do anything to fix it we have to have some notion of what it is we're actually fixing right and it came down to three properties the first I basically already started to talk about right it's that programming is unobservable I make a coat or I make some change in code unless I manage to execute that code path you know in manual clicking around on a screen or whatever I have no idea what happened and as I'm making the changes if I make a whole bunch of them I have no idea what the effect of each individual change is much less the effect of the entire aggregate of changes and this is a lot of what we were trying to address with light table was trying to make programming more observable but there's another problem too and that is the programming is indirect so if I'm let's do something fun if I'm describing a drone right and I have this thing hovering over here next to me if I want to make that drone do something using pick any language you want right I have to first translate it into that language I have to first translate it into the computer right I have to I have to think like the computer in order to get the computer to do what I want is this very indirect mapping and if you look at the vast majority of errors that happen when people program they come from actually relatively simple translation errors right as this indirection that causes a lot of friction so when we were doing that that study on visual studio we saw this over and over again where somebody had the right solution just along the way they just did it slightly incorrectly and again it wasn't that their logic was incorrect or any of that it's just literally the mechanical process of translating that logic into Python or C sharp or whatever is not easy we're not very good at it as human beings and the last one and this is something that career programmers really latch onto when I talk to them is that programming is incidentally complex what I mean by that is that the vast majority of what we do when we program has nothing to do with the problem we are trying to solve right a very very simple demonstration of this is I like to pick on how long does it take to set up a dev machine from scratch right I watch people take the better part of a week doing this at Microsoft that's just to get code run right that's like not even trying to do anything interesting yet that's just to get started to start your problem right and so when I say that that programming is incidentally complex I mean that we spend a great deal of time solving the problems around solving our problem so to give it a simple example of this to sort of demonstrate exactly how hard programming really is right we tried to come up and we did this talk a while back where we went around and and had a bunch of people try to build what we thought was one of the simplest possible programs you could imagine a normal person wanting to build which is a button on the screen that counts the number of times it's been clicked right and so you know I can click the button it says one and two and three and that's it right if you think about this like what it takes to build this little button we'll assume you already understand HTML which is a big jump right but we'll assume you won't know enough to write this little bit of HTML here and then it's not so bad right it's just six lines of JavaScript well let's unpack exactly what's in the six lines of JavaScript so first off you have to understand the notion of identity which is arguably one of the hardest problems in programming even though it seems like it should be something very simple you know I have to tell the computer hey that button that I just put on the screen I'm still talking about that button I want to work with that one right you then have to deal with scope right this this again six lines of code has a function in it which is creating a new scope and things are weird in this sort of world things that you define in there don't escape but if you talked about them above there they're ambiguously escaping you have the notion of mutable assignment vast majority of programming errors are the result of mutable assignment you have closures something that still most career programmers don't understand correctly you have a synchrony which is probably unquestionably the hardest problem in programming at this point you have objects and properties and the Dom that six lines of code contains the context of years of work right you have to have spent years to understand what all of those things are doing yeah sure somebody might be able to go to stack overflow you know copy some of this out and tada I have a button that works but that means they can't change it cuz they don't understand it right those six lines of code you know I joke here like too long didn't read for six lines but that is kind of the reality right programming is hard question is does it have to be hard right is this is this sort of intrinsic my standpoint is no but this is where we ended up with light table right we had taken these this kind of language we took JavaScript we took closure we took Python we're like well if we remove the feedback loop programming won't be easy now right because you don't have to play computer in your head but this didn't change all of this is still true right all of this is still necessary and so as a result of that it you know didn't make things much better and that's sort of this sort of like fundamental problem with trying to make it more accessible right as programming is still programming no matter how good of a tool you make for it and the problem with that is that these languages and these tools were built around the idea that we were going to build applications right that the end result of programming is a little icon that you double-click on your screen and does one thing and somebody writes it and they give it to you and that's that but the reality is what we really want when I started to think more about what it meant to make programming more accessible it wasn't about making applications we struggled with this for a long time right we're like oh well it's going to be building website so we'll make it easy to build websites right but it's actually to do something far more fundamental right it's just to think like if fundamentally if you go back to the beginning of of you know the 1970s when when a lot of what is modern computing came to be the idea back then wasn't the single purpose appliances right the idea was a tool to think with and if we're going to make programming more accessible ultimately what we are trying to do is make a thinking tool not necessarily a tool for building large-scale applications right because at the end of the day the computer is the greatest thinking tool ever developed right if I know the magical incantation necessary to punch holes in this card I can get the computer to think for me right and that is a hugely hugely powerful thing you know when we went around pitching this to folks we labeled it a superpower because that is fundamentally what it is it was one of the only times that given a computer you can start with nothing else and create something of immense value right just by knowing how to punch these holes and so that would that's what sort of ultimately led us to this project that we're working on now that projects name is Eve and it is this idea okay I started with with visual studio trying to make programmers better realize that the tools are actually not all the problem right it's look like the model of programming itself is a problem got to light table like hey the feedback is now there so that makes it observable but it wasn't quite the way there and it was still a tool right we're still approaching this from a tools perspective Eve is what would happen if we tried to solve this problem from the ground up right ignoring basically everything that exists what would you build if you could if you could have read if you could basically imagine building programming today right so the first net first approach to this of course as a programmer was to take all of the things we thought were good about programming and all of the these sort of trends that were happening like functional programming immutability and all of these things we'll put them all in a pretty little package and that's just going to work so the first prototype of this was the thing that I showed at at Strange Loop which is a which is a software conference no is it not playing oh there goes it was a software conference that's sort of like deals with a lot of the more interesting sort of avant-garde ways of thinking about the world and so the idea was okay we'll take functional programming in a mutable data and we will put it in something that looks a lot like a jupiter notebook right and we'll make it so that you're like literally working with the data right I like right-click on a thing and I tell it what I want to do with it and you can see all results and don't get hung up because it doesn't look super great at this point I built this prototype in like four days but it was really interesting because it took a problem that would normally take quite a long time better part of a day for example so this was taking a bunch of data out of doing so usage data out of and then doing some some math on it and then getting a graph out of it right and all this again was done by just sort of like dragging and dropping and right-clicking on some things and there were a bunch of interesting ideas in this one is the idea that you're always kind of working with the domain itself so you'll see here in a second when I want to write out some math I actually do it in math and by that I mean mathematical notation zoom ahead here I'm apparently being slowed right so you know I want to sum over all the X's and divide it by the number and today I have average yay so the idea was write like you're working if you're working in a domain let's say for example I have my drone example again right if I'm working with a drone there's a drone on the screen right instead of describing a set of operations that I'm performing I just move the drone around right and it's capturing what those steps were and then I can parameterize them by data or like get a chart or whatever I'm trying to do right so this was interesting and we took a much better much prettier cleaned up version of this we're like this is awesome we can build all sorts of things with it later in the video I show I'll skip ahead I feel like building a website this way where you know you just have these tree of functions that do stuff and it was interesting and cool as a programmer it was very exciting and I happen to be at the opening of Brett Victor's laboratory CDG in San Francisco and there were all of these non-programmers brilliant brilliant people physicists and mathematicians and all these people there don't know how to program and we set like I said a much better version of this in front of them and like try it right we're like we were excited we're stoked this is we're done this is gonna be amazing and actually the reactions were really positive people liked it they they thought it was really interesting they got why they would want to use it and then much to our horror we watched them not be able to do anything in it right and I will never forget the reactions we got which was they were trying so hard right and and they were very well-meaning they like they genuinely wanted this to work so they were like willing to do things that most people probably were not really willing to do but there was this thing that surprised me this thing that they struggled with out of all the things that I expected to find out in this the hardest thing for for the non programmers to work with with scope so in functional programming you end up with these deeply nested set of functions right everything is about this magical tree of functions that you're going to call and the the reactions that we got were really obviously bad so like somebody would look at it and go you know they had some data up at the top of the function stack and they they had a some calculation they wanted to do at the bottom of the stack right and they're like I want this down here and you're like I don't look at I mean oh well you can't I mean you'd have to pass that through all of these and you can't really do that and they're like mmm I don't understand it it's right there why can't I use it here right okay fine if I can't do that can I use this up there no you can't really do that either we'd have to thread it through going back up right it was like this really I I no longer believed the arguments I was making because you're like trying to understand like well scope is really important right like you need it for functions and functions are really important and I don't know that's like something we can't do right and so we try to do a bunch of things we try to make this nicer right like okay well we'll make it look prettier and we try to do a bunch of things to make the scoping thing less of an issue so there was no variable shadowing we had a much simpler version of scope we do arrows we colored things the same we try to do all of these different things nothing worked it seemed like it was kind of like just this fundamental problem that was really hard to get over and so that led us down this very odd line of questioning which is well if if our first attempt at this what the industry seems to think the future of programming is is like I said functional and immutable and yay if that's not right and scope is a kind of a problem which basically encompasses all the programming where do we go from here right like what does it mean to remove scope from programming so if you think about it well scope can't functions so that's already scary to begin with it also is that when you're building up these like nested data structures that you have there they're like creating scopes right you're like starting to worry a lot about where something is in this tree of data and you don't want to duplicate because you've put that data somewhere and you put it in another place then they're gonna be out of sync and you have all these problems there's another version of the scoping problem so you can't have functions you can't really have data structures if you don't have data structures you don't have anything that can keep things in order anymore so you don't have ordering so if you think about what programming is if you asked a career programmer what programming is they would probably say an ordered set of steps typically executing functions over a set of data all of that falls away if you remove scoping and so we started to look into well has anything ever worked this way right and there is at least one massively successful example that works exactly like this and that is Excel right so Excel has a single data structure but you don't work with the data structure directly it has no scoping if I can see a value I can use it anywhere it has no inherent ordering it has scalar functions but you're not like nesting these deeply right you're writing out basically equivalent of a mathematical formula and it never goes deeper than that so it has all these properties and that is a fantastic result and also a terrible result but a fantastic result first it is a fantastic result because 800 million people use Excel every single day it is unquestionably the most accessible thinking tool available to us and people have done amazing truly amazing things with Excel I I was telling the story at lunch that I had this opportunity to talk with some folks at a fortune 500 company right and this cruise line and they have this spreadsheet there was ten gigs in size they had been built up over a decade no professional programmer had ever touched the spreadsheet but it modeled the entirety of a fortune 500 companies business they could have nuked every other system and run it off of this spreadsheet right this is amazing like totally utterly amazing and so that's great now at the same time Excel is this is the downside Excel wasn't meant to do that kind of thing right so that you know they were joke around they're like yeah if we made any changes to the spreadsheet we had to go to lunch because it took an hour and a half for it to propagate right and Excel is constrained right as a two dimensional grid of numbers if your problem is not easily specified as a two dimensional grid of numbers kind of out of luck so we looked a little bit more and we found that there was one other case that worked this way again massively successful and that is relational databases relational databases have the same set of properties right they don't you know if you can see a value you can use it anywhere if you can see a table you can use it you know barring access rules basically there's no inherent ordering either to the code the execution any of it right even the data has no inherent ordering you order it as you will and again no like deeply nested functions or any of this crazy scope stuff going on you have basically functions and scalar values and that is a great result again certainly not accessible like Excel but it is it solves the other side of the equation it is a flexible thing right the world runs on relational databases at the end of the day basically every bit of data ends up in one of them and we have built games and websites all the things you couldn't imagine really building in Excel so we discovered this like hey relational databases is an interesting case let's see what people have been saying about them recently right there's got to be stuff other than Postgres out there that we can look into and that's where we ended up on research out of here actually Joe Hellerstein lab the boom lab had been working on a language called bloom and they were working on that language to do distributed systems the idea behind it was that it was a query language so a little bit like sequel but it was based on data log and you can if you know what Prolog is it's kind of like prologues little cousin and it does a lot less and it's a lot simpler and is able to be a lot faster as a result and they were building distributed systems so they're like reimplemented hadoop right and we're like oh that's really amazing because it was orders of magnitude smaller and all these great things could we build a website this way right like could we take this maybe stretch it a little bit think about it in a slightly different way and how far could we take it so I built this prototype this is ultimately actually the prototype we raise money on and it's a very interesting thing because if you read what these little things do we had an intuition which is that prologue right or data log or any of these is written in predicate logic and predicate logic is pretty hard to get your head around but if you think about it it kind of looks like fill in the blank sentences like those mad lib books we had when we were kids right it's like put a color here and if you squint and you think of it that way if you think of these predicates sit that way and you think of horn clauses as like when you see this thing do this thing you get this kind of very natural way of thinking about it so this is the incrementer button right that little button I was talking about before say like well when I'm setting defaults remember that the counter is zero and when the counter has a value draw this button and when that button is clicked change the counter is whatever its current value is to the current or to the value plus one right and so like I said we built you know interesting things out of it we always build two new MVC because that's what everyone builds but sort of building interesting things out of it and we started to realize this crazy way of programming you know that has no data structures has no inherent ordering when I built this prototype originally I didn't even have aggregation which is interesting sort of made us realize that a lot of the things we think are fundamentally necessary for programming just aren't they just you don't need them at all actually and sort of as an interesting sample this not only yeah we built this website and you could say all well that's a toy or whatever the IDE that it was written in was written in it and I know a thing or two about building IDs at this point I built a few you know visual studios 55 million lines of code light table which does far less than Visual Studio and was designed specifically to be as small as humanly possible was over 10,000 lines of code this was 600 right we're talking about a totally different way of thinking about the world not only was it 600 lines it was written in a week in a language that had no tools that didn't even have aggregation right like I had bugs in it and we still managed to write this in a week and so that is a tremendously awesome result that you know we can take some of the things that that Joe's group had learned about distributed systems and apply them more generally to these other places and if you squint at this it starts to kind of look like one of the Holy Grails of programming right it kind of starts to look like an executable an english executable specification right I read it to you and if you think about the the description you would give of the increment or button it's not very far from this except maybe the wind setting defaults part of it right and that's obviously pretty surprising and and like really interesting right again this is one of those things we've wanted for forever but we found out that actually this way of programming isn't that great the whole I I you know a lot of people still think that the English specification is is like the future of programming if we ever get there we've made it and we're kind of done and I no longer believe that's true the reason why after having programmed this way for a little while is that if you read an English specification every line of that English requires you having understood and internalized every previous line that was written right if you think about a big part of what what makes programming hard right or a big part of dealing with complex system is being but box parts of them up right like I don't need to know about this I don't need to know about this I can just let go with this and these English specifications required you to basically understand the whole thing all the time and yeah they were shorter so maybe you could reasonably do that but when we actually tested this out with people and sat them in front of it they were like I can read these programs I get what they're doing no one was successful in writing one right which is kind of interesting and yeah you mean you could argue maybe it was our tooling maybe it was our strategy maybe it was whatever but I think it's probably actually more fundamental than that like I said I think it has to do with this context problem that every line requires the context of all the previous lines and so we get okay let's try something totally different we know this Excel thing works what could we make it look like Excel right and so we started to realize that these operations you could perform visually on tables because that's basically all that was going on here and so in this example it's gonna be a little hard to see so don't worry about it too much but what I'm doing is this is the first time you've interacted with the outside world it sent us a text message and so we did we have the set of buttons down here at the bottom which you can't read but it has our names on them and if you clicked one of those buttons it was going to send that person the text message and the way this worked was that there were these tables inside of our system that you know had like event information in them and had the people in them and then we would join those two things together and now you can see every time I click on one of these it has the information for Chris or Jaime or Rob and has a fake version of our phone numbers but we actually did do this with our real phone numbers at one point so it's kind of like writing this thing in Excel again with some more interesting abilities than Excel usually has right we can join things it had basically the properties of a relational database and this was this was neat it worked for some definition of worked it allowed you to do relatively simple things but we found a bunch of problems with it for one once you had this nice little table you lost all context and where any of this information came from which when you're building things is actually pretty important information to know you just have these values they just have kind of like random names but it did kind of prove to us that you could start maybe treating this as a very different problem maybe instead of thinking about like writing code even if it were English code maybe you could think of it as like operations in these tables and you could do it very visually and you could do it almost essentially just mouse-driven right and it could look like excel we sort of ran into a bunch of roadblocks here like I said just like a very simple example of this is not knowing where anything came from but as you started building up these more complex tables you you realize that people don't really think of the data this way they're like they have to think about keys a lot right and like joining on the keys and it was still pretty hard it wasn't it wasn't the right solution and it was shortly after this one they were like well guys I don't know yet about how to do the whole like make this thing accessible but can we at least prove that it's a good idea right that the foundation of this is a good idea because we had essentially broken this problem up into several problems the first was could we find a version of programming that was very simple right a brand and by that I mean of few parts and that version of programming has to be something that I could use as well as it's going to work on excel like problems too right so had to be a general-purpose language but I needed to be very simple and so we really really liked the the stuff out of the boom lab and we had built some interesting things but we wanted to deal with it more we wanted to really try and develop some stuff with it and see what happened and so we built this prototype which given how much time do I have not too much time I'll show you very very briefly what this looked like so we had this thing don't worry about what this looks like but so you know here's a clock right we can zoom in a little bit and everything in this language like I said it's just like kind of like sequel basically which is kind of this weird way of thinking about the world where all you're ever doing is querying and and specifically actually you're building views in our case which are queries that are always up to date more or less and so very first thing sticks a row in a timer table which generates a timer for me and that's what runs this clock and then down here we have this line which pulls in from the time table which is what that timer values into and then we aggregate over that timetable to get the latest time and then we just turn it into hours minutes and seconds right and then we draw our clock and this is just me being bad at math and then they had this little language down here that allowed you to describe UI right and we never what's interesting about this language is you never describe how things change basically all you're ever doing is like asking a question and formatting the result of that question any but all sorts of crazy things this way yeah we you know we built the clock that's interesting of course we built to do MVC right again I have to do NBC we also built a Turing machine just to be sure so that you know is in there and what's interesting is once again we bootstrapped it so everything you saw being rendered on the right hand side all of those tables all the UI even so this thing was actually pushing processes distributed so like it would run your program in another thread and then subscribe to that other thread and pull data in this entire little thing here was written in two hundred lines this is all of it right here so again this is like totally like crazy way of thinking about the world but it kind of works right and so it was from this prototype that we started realize okay we don't yet know how in the world we're going to present this to people but this seems like a pretty sound model to build off of right there are definitely problems that we have to figure out you know there's performance and interesting things there although surprisingly if you reduce your language to only operations on a b-tree which is what a query language ends up doing it ends up being pretty fast kind of by default but there's still lots of interesting work to do there but the foundation is solid the foundation can express the kinds of things we want to express and we just have to figure out how to show it you know I get back to my presentation so we try it again this time weird I don't know why but we ended up going back to the excel thing but this time we had something a little bit more interesting which was you can actually build UI and this was the first time that you had like a drag-and-drop UI builder and I know that's a little hard to see but I dragged out a button there right and we're gonna build the incrementer example and I'm gonna give my button a name we'll call it incrementer and there's this little button down here that says get the clicks for that button which is just a table everything is always just a table right if data is going to come from the outside world it gets shoved into a table if you want to do something to the outside world we shove it into a table and something and out there kind of looks for it and does something with it is every time I click I got a I got a row in this table and then the incrementer button is just counting the number of rows in the table right and this had that excel thing going on so you wrote like an excel formula in here and you said count them and then you could just drag that column onto the button and that was now bound now it's sort of like data bound it to the button and yeah I was asking me what to bind it to bind it to the text and an incrementer button so what a problem that took we would take a professional programmer easily 30 minutes to do assuming they knew what they were doing and if they don't who knows how long could take 30 seconds right like we could be much better at this if we thought about the problem slightly differently that one again suffer from the same problems that I described before which was that you lost all the context when you're building up these tables and it didn't really fit the model very well so we tried this other thing we're like hey the UI builder thing worked out pretty well let's clone Foursquare let's clone a production-ready website and see what that looks like and we built each time right we build a new Eve to try and do this trying to try and understand it we test it with people this one was good enough to build the production-ready clone of Foursquare it was abysmal doing it oh it was really really bad although not because of the UI editor that worked out really well but the way we wrote queries in this was really awkward we tried to mix the - so we had the tables right and then we had the the information basically the sources of all the information from the tables up at the top and it kind of looked like a equally looking language you were just bad it's hard to explain exactly what was so bad about it until you use it and you like fully realize exactly how bad it is I don't have enough time to show it and I don't want to be ashamed so I'm gonna skip that one but it was still interesting it was it was the first time we did not have to write any code to build a real website right like you could do this purely without typing anything more or less which was interesting again but it just it still wasn't right so the next step after this was what we we initially released so we're all open-source and this is what we we pushed out when we open sourced and it was based on this intuition that well all of our attempts of writing query is not in a query language that a programmer would understand have kind of been crappy but we had this intuition that when we passed around queries to each other we drew them on the board and we drew them as what looked like entity relationship diagrams if you know what those are little bubbles with you know lines between them and we realize well if we draw them that way to explain them to each other could we make them work and we hesitated going in this direction because we know lots of problems with this kind of visual programming and we had even built the you know the boxes and arrows version of this long you know early in our process and realized this is not going to work but this is a very different property because what you're describing is not a set of steps or a data flow you're describing a shape you're saying I am looking in the database for a shape right and specifically this is a famous shape it's known as the three click query or the triangle of friends basically a friend was a friend of a friend was a friend right this happens to be a notoriously difficult thing in databases to optimize for and you'll notice it has this wonderful property I asked for a triangle so I represent it as a triangle this is one of the first times that people seem to understand relatively complex queries without us having to explain much of anything to them a very simple set of things and they were like oh okay triangle of friends it's a triangle right and then you know here's a slightly more complicated one where you're getting the you know you have a bunch of departments and you're trying to find out what percentage of the total cost each department is there's a bunch of aggregations going on in here and then you're dividing those to get the percentage and this was really interesting because like I said it was the first time people could understand it it didn't suffer from the same same scaling problems that dataflow like visual dataflow languages do because your shapes never get that big right it's not like you're describing the entire flow of a program which could be you know of infinite size basically these were you're only ever asking for some shape and it looks roughly like this and so it seemed to scale halfway decently it was really hard to build systems in it though and we learned that this focus on individual queries and results wasn't really the right thing it sort of it took you away from what you were actually carrying though which is the data right you're always asking your question to get an answer back and this focused a lot on the question and a lot less on the answer and these like units ended up creating these like silos that you then did a lot of back and forth with it was a lot like how you deal with you know if you're building a program and you have a bunch of different files right and you're like going back and forth because the functionality you want is across four or five different files well you have the same problem in this but exacerbated where you know it's like only looking at one function at a time right okay so we zigzag because that's what we did and we tried hey let's take the Madlib looking thing the fill in the blank sentence thing let's put it in a notebook like the very first one let's focus a little bit more on the answer and let's make it a little bit more visual and let's see what happens three with this and this was interesting because this one would allow you could actually like talk to it so we did this I put this I got it running on my phone and phones have dictation on the now as you can literally just like tell it a query right and it would just give you an answer back which is really interesting and cool like that's a compelling place to eventually be but we found that unlike in like the Jupiter case we're kind of like writing out the set of steps you got to an answer for we're talking much more like these are alive right they're changing they're there they're not captured in time so the linearity of a notebook is a really bad solution to this problem because it implies that this then this then this then this which is not true in this world right so you might do something at the bottom that would totally change the first thing and that didn't fit the didn't fit the model very well and also it still feels like programming you're still kind of doing this joining stuff you had to know the names of the things you were looking for right not the names of like Chris but like the names of the relations is friends with or you know has a house right which again is a hard thing because who knows what they are if you didn't write them you kind of have to guess and yeah we could build tools that help you navigate that but you're talking about lots of them it's just doesn't matter what tools you have right okay so we love the foundation we still hadn't figured any of this like UI stuff out we tried we had built basically 30 versions of Eve at this point right and we were like okay we're clearly doing something wrong what what assumption are we starting from that's screwing us up and I think you're the assumption was and once we threw it out it like cleared things up a bunch was that we were still approaching this as if we were building a programming environment right and the idea wasn't - it wasn't really to make programming environments right the idea was to enable a thinking tool and a thinking tool doesn't have to look like a programming environment so stop basically and where we are now and the thing that we've decided to commit to because it's tested extremely well and we're extremely excited about it it doesn't look like programming at all not even a little bit so I'm going to show it to you a little bit so we have this box and it's kind of like a search box it's just a really fancy search box more or less so I'm in the system if we look up Chris Granger there's not a lot about me here I'll make it a little bit bigger it's not a lot about me but it says you know I am an employee at Kota wah that's the company that we run I have a salary of 20 which is way too high we'll change that I have a salary of two so this is actually kind of like a wiki interestingly enough if you know much about wiki's this is what's called a semantic wiki with some changes to it that make it a lot more powerful than semantic wiki's typically are but the idea here is important one of the things we found every time we focused on the excels like way of representing things was that people don't think of their data as like tables a lot of the time right a lot of the data that we really want to put into a system for thinking yeah some of it is going to look like a table you should be able to do that but a lot of it doesn't a lot of it's stuff like random stuff about Chris and I'm just going to like write random properties that I happen to know about him at the time write that like you know I am and I have a salary of two and you know you know his height is really short right right and now tada that data is just available there for me now and I can start to ask questions about it so what is Chris Granger salary well it's - what about his height the thing that I just add hey short right so this has this very very fluid way of working with information what's interesting about it is again one of the problems with this navigation thing I was talking about before right you have to know the names of all these relations and all that stuff well in this case they're just links right like here are all the things that I'm related to it even tells me down here that I'm related to another thing called engineering and if I go there I see the engineering card and here all the engineers at Kota Hwa and that's interesting and I can like dive around in these just like you could on Wikipedia right and that's actually a really interesting thing because a we know Wikipedia works be if you could imagine a wikipedia where all the data was available to you to ask much more complex questions that's a pretty compelling idea so for example we know that there are departments and I can look and it says oh there are two departments here this is just sort of generated by the system right if I add a new card that says it is a department order to show up here but let's do something interesting with it like let's sum the salaries per department so if you think about what just happened there this is important there is no direct link between department' and salary' departments don't have salaries employees have salaries right but because of the way the data is structured we can trivially figure that out for you so that you can say it in this very natural way right I don't need to say well first get all the employees then link them to the department then group them by that and then sum them and actually if you want to see we can actually see exactly what it chose to do I can pop it open and it says oh it actually just gave you the process I just described gather departments gather the employees related to those look up their salary group them in someone but instead you just ask this very straightforward question right and you can see it gives us back this nice data table here and and there's another important thing it shows you exactly how it got that answer so this is something that sequel will leave you in the dust with when you start to do things like grouping you start aggregating all all the aggregates are lost right all the things you aggregated over are gone one of the things that's amazing about this way of working is that we know where every aspect of the information comes from we have complete and total provenance information and so we can show it to you this is exactly how we calculated the number 16 here are the people that went into it okay so I said I mean that's cool we can ask questions get an answer but this isn't programming yet you need to be able to format the results into something interesting that's where you get programming that's when you get change and stuff so the way we do that is just by adding a card so we call these cards and we want to do is we want to say this is like a placeholder we want to take the department for each department up there right and we're gonna say something new about that department we're gonna say well that department has a total cost and that total cost is equal to the sum that we have there and you'll see now we have this little card down here and nothing looks like it's changed but if we go back to engineering you'll now see there's a new value down here at the bottom of this car this is engineering has a total cost of 16 I can now compute over that so we can start to say departments with their total cost right it's just new information and so this is like almost like a wiki where you can ask a question and then generate new information as a result of that and that is enough to build anything you can build compilers this way you can build websites this way you can do all sorts of crazy stuff what I think is the most compelling is that you don't need to build an application to do anything here for the most part you could just work in this right you could just sit here and continue to ask questions and set up these little rules that you have to do very interesting things so for example let's go back to Chris and let's make my salary crazy again just way too much money right and if we go back to let's get the total cost of departments you know it doesn't really matter how I write this for the most part total cost of the department's you'll see it's 34 and actually you know what let's let's change this around and we'll say that departments with a total cost greater than 20 all right so this is actually a full query language you can write all sorts of crazy stuff in here and what we'll say is that those departments are this department is overspending right and now if we go to engineering did I not do it correctly I forgot what I called it and not to you're too important all right departments with a total cost greater than 20 and this department so ever spending if it doesn't work this time we'll all right good there you go now this department is overspending and now I can again start to ask questions about that get all the departments are overspending send their heads an email right and the interesting thing is all of this stuff is live so if I come down here and open another one and go back to Chris Granger you'll see my engineering has a total cost of 34 that's because I have this really high salary but if I go and I change it back to two all goes away it's 16 right this is all alive I never described how things change as they changed it's just abused over the data and so I was going to show an example here that thought was funny so there's the Apple the recent Apple Keynote they showed a thing that got a lot of people really excited so they showed on the Apple TV you can now talk to Siri and have it find you the example the specific query they gave was episodes of Modern Family that contain Edward Norton right so we can do that so we can say episodes of Modern Family with Edward Norton there you go that happens to be the only one we can also say well how about without Edward Norton how about we count them right this this way of working allows you to just very very easily very fluidly just start to ask questions and I think that the the craziest realization for me and all of this has been if we took a step back and stopped thinking about this as programming for a moment right we managed to come up with a thing that doesn't look like programming anymore it's just asking questions and formatting results and that encompasses all of the things we want to do and that is like a like an amazing result to me that is that is a fascinating thing and there are still lots of work to do here right I'm not saying we've done it and I have no idea really what you know programming is going to look like in ten years but my hope is that you know whatever programming does look like that it looks nothing like the programming we have now right the program we have now like the last thing I want is you guys who are trying to cure cancer who are trying to understand the cosmos or whatever you're doing have to worry about Unicode or have to worry about event ordering or you know like these ridiculous things nothing to do with the amazing stuff you were trying to do I don't want to look like that at all right because at the end of the day like I said the real goal here is a thinking tool and that is what we have to get back to and it's in again it's not a thinking tool for just me right it's I want to remove the cards I don't want you to have to punch the holes anymore and I want it to be able to think for us all so yeah that's my spiel

Channel: Coding Tech

Views: 49,645

Keywords: programming, software development, coding, web development

Id: zQo4jUVWI_I

Length: 56min 10sec (3370 seconds)

Published: Fri Nov 17 2017