GopherCon 2015: Robert Griesemer - The Evolution of Go

good morning everyone so as one of the first three go team members next to rob Pike and Ken Thompson I was invited to talk a little bit about my views on the creation of NGO and its progress so far and its future this is naturally a more personal talk and others may see things differently or remember things differently so keep this in mind but let's be clear up front go wouldn't be possible without the rest of the go team and certainly not without the open source community so it was a privilege and also an honor to be able to contribute from the very beginning with that disclaimer out of the way let's get started the evolution takes some strange turns sometimes so let me give you a little bit of personal background I've had an early interest in programming languages and it really started when I saw the first report on the programming language Pascal here there was a description of a language that was concise disciplined there was actually a method behind the design there was quite different from the language manuals that I had read at that time which was mostly basic manuals and perhaps some you know pocket calculator programmable pocket calculator manuals so this interest only deepened when I was able to study at ETH Zurich in Switzerland under Niklaus fear - is actually the creator of Pascal and several successor languages and they actually had a direct impact on go as I will try to point out a little bit more here it's interesting we had a talk the last talk yesterday went into the same direction I'm trying to not have too much overlap here and there is a few Corrections perhaps so at grad school industry programming in industry sorry when I left grad school prom in programming in industry felt quite like a huge step backward in at university we had a system that was built there using a language that was built namely all brawn and it was extremely productive and so when I started programming in the industry which was essentially C++ programming I just didn't have the productivity anymore so I really wanted to go back to this productivity and so as time went by over the years I spent a lot of time thinking about programming languages on how to make them better but actually it is really hard to make a good programming language in fact it's so hard it's it's just a misnomer that we call this thing software so but at some time at some point you know after 15 years of programming in C++ there comes a day of reckoning you have to figure out what to do with your life you know are you going to sell out or bail out and so we decided and I decided to bail out and that was not a hard choice and I feel extremely lucky to be on board we both Robin can so the beginning of the NGO project started pretty much as a result of the frustration with a long C++ build and Rob Pike had talked about this before but what was important was that we had a very clear goal in mind we needed a better language for the things that we do at Google I also had a personal motivation I wanted to have that clean and small and compiled and compact and type-check language that I had worked with many years back and I wanted to get back into the fun of programming or in other words what I really wanted is I wanted Lego bricks which which with which I could build stuff and I mean by that the elementary LEGO bricks not like the more fancy Lego bricks which can only be used for one thing you know like to cover like a car or something those Lego bricks are actually not orthogonal anymore when it comes to the design of you know the LEGO pieces and by the mod by modern features I mean things like modules or packages and things like closures keep in mind C++ is only getting packages or modules about now same is true for closures and the closures in Java are best in approximation also both these languages are incredibly complex for instance the latest deepest plus standard is about a thousand pages in in description so here's a little anecdote from a from Google so it's not uncommon that an engineer runs into a problem with C++ and you know she sends out an email to the wider group to figure out what the hell's going on and so over the course of the day lots of people chimed in and sent her elaborate detailed answers all of them are different eventually at the end of the day the local C++ guru the one that wrote the book about the subject you know answers conclusively in a two or three page email this is completely wrong I mean here it's obviously the programming language that's getting into into the way of things and you don't even have time to think about the actual theme that you should be working on so we had some pretty good ideas of how to how to address these issues that we saw and we had quite a bit of unpolished thoughts about the rest and there were these we were three people with quite a bit of experience and we thought you know we know how to not do things and then of course there's always the lure it's going to be fun to design a new language and it's going to be easy what could possibly go wrong so we had a few guiding principles and the first hour probably simplicity safety but at I mean memory safety unreadable II a readability are paramount and I cannot repeat that a Canada under under state that so let me say it again simplicity safety and readability but I would like to also point out that by simply the simplicity for language design doesn't necessarily mean that other things have to be as simple and I want to refer to the talk by Kathryn yesterday where she pointed a little bit in this direction just because the language is super simple doesn't mean that your final product has to strive for the exact same thing because sometimes that's not possible sometimes the thing that you're building is not simple by design of by definition so there's there's there's levels so we were striving for orthogonality in design and the reason for that is pretty clear if you have orthogonal features then they don't interfere with each other and they're more easily combined and we wanted the language to be minimal in in the way that there's only one essentially one mechanism to write a particular one way to write a particular construct so that takes away a lot of decision-making when you actually write your program and there's a few exceptions and goes most notably variable declarations and perhaps the even switch statement where there's some overlap these are mostly for programmatic reasons so we wanted to make sure that the things that we wanted to write that were of interest were easy to write but you didn't want to make everything necessarily possible for instance we didn't want to make make it easily possible to write lock for your algorithms that's something that is in the domain of you know experts and most people never have to do that so how do we get started so it turns out there's not much good literature on language design but there's a few papers and there's two papers in in particular both are by Tony Hoare they're a little bit dated but they're still quite on target and I recommend them to everybody who is interested to read more about the subject the second one reads like a Woody Allen movie and I promise you it's just about as much fun even though doesn't talk about sex so Cain Rob and I got started initially with a brainstorming session and the whiteboard just to scope out sort of the rough ballpark where we wanted to go I think it was a Friday afternoon and there was a weekend ahead of us and there was a lot of time to think so at the end of Sunday I had written down everything that I recalled from our brainstorming session and lots and lots and lots of stuff that I thought about and there were my ideas simply because I want to make sure that we are going into the right direction so here is a here is sort of a summary of actually it's not a summary this is really the first paragraph and then this male is very long it's like two or three four pages long I'm not going to go into the details but it's interesting when we're looking back that many of the ideas of this very first day made it in to go from the obvious such as you know the leading keyword notation lots of syntactic cleanups to you know expressions of with only five levels of binary precedents as opposed to the 14 in C++ to explicitly size types to avoid problems when you pour to different platforms to the less obvious such as packages and imports methods with explicit receiver types sorry receiver parameters and interfaces and concurrency but many of the concepts were missing and even more of the ideas that I had written down and we thought about didn't make it at all however we were off to a pretty good start but this was not clear in the beginning and obvious at that time so the origins of things always casually blurred over time and something that actually took a long time to conceive evolved really later is imagined as this singular invention you should keep in mind at all times that most ideas come from previous ideas and I apologized to Alan Kay he seems to be quite popular at this conference actually a lot of critics say there's nothing new in go but they're actually missing the point because if you're designing a new language you really should not invent you should consolidate features that have established themselves for a long time so consciously or not a lot of the work we had done was exactly this we would take existing features and mechanisms distill them down and maybe create a new syntax for them or adjust them as we deemed appropriate and by having three people involved in the process we made sure that nobody's you know favorite idea sort of got out of hand so we had our own sort of checks and balances in place it was not always pleasant though to get your favourite idea decimated to nothing so where are most of these pre-existing ideas coming from and it's really informative to look at the history of programming languages and most notably we don't have to go all the way back here but Algol 60 is really the starting point of everything and again yesterday is last talk by B G already mentioned some of these connections he had to cover it necessarily from the outside I'm trying to illuminate the background a little bit more in detail so about 60 was both a consolidation of what was learned before mostly from or most notably from Fortran enlist but it was also at the same time revolutionary in its forward-looking design in fact it was so forward-looking that horror said here is a language it's so far ahead of its time that was not only an improvement on its predecessors but also on nearly all its successors and he probably meant outlaw 68 at this point so alcohol seeks to introduce the block structure nested and recursive functions procedures type declarations static typing you know that the usual keywords semi colon separated statements and so forth there's a few fun facts about alcohol McCarthy actually suggested that they should introduce recursive functions and little did they know what they were in for because that was fairly new mechanism at that time and it turns out that the key word for actually comes from much earlier from 1952 from a description actually for algorithms which was called super plan the German word and in that description the key word was not four but it was fuel at a German word for four coincidentally just two years before outlaw 60 actually for alcohol 58 backus naur who worked on Fortran also came up together with peter naur with the packers in our normal form which is now very much the standard notation for syntax sometimes it's the extended backus-naur form I think oh it turns out that actually the algal successors are really the ones that have direct impact on though namely Pascal and C Pascal is a successor of alcohol created by niklas weird it shares many of the same features it introduced semicolons separators it uses the all uppercase keywords which is now makes it really hard to read I introduced left-to-right declarations principle duck structured data types it had even a notion of we declared functions very much like we have in go then on the other hand on a different continent C was developed and it's actually quite similar to Pascal in many ways structurally at least instead accept that there's curly braces for blocks semicolons are not separate are not separators they're terminators and declarations are not left to right instead they mimic use and there is there is much better support for arrays and pointers which is one of the things that was missing in Pascal and really led to problems sometimes I want to look a little bit more closely at some of the Pascal successors so there's several of them first of all there is medulla and then eventually modular to model a' to is important because it introduced the notion of separate compilation via modules and by separate I mean separate not independent as in C where you have you know two files that you can compile independently but there's not really much checking happening except through you you basically have to repeat declarations that you refer to from from the other file via and include in modular two you have an actual import statement a few years later Niklaus weird worked on Oberon which is a successor of modular 2 and where he tried to distill the essentials of mod litter into a much smaller language so in many ways Oberon is kind of a forerunner in this in this sort of approach to to radical simplicity to concentrate to the concentration of the essential at least for a statically typed language Oberon introduced the concept of type extensions which allowed basic form of object-oriented programming but it was a little bit cumbersome in that respect so my adviser HP Mesenbourg at that time experimented with that language and derived two dialect which he called object Oberon which introduced classes and methods and over the years and then eventually together with Niklas weird that dialect was a little bit refined and we ended up in Oberon - and the interesting thing is about Oberon - is that it introduced methods in a very similar notation that we now use for go so let's have a look at a piece of code in Oberon - so here we have a module which is really a package called trees and there is a tree data structure and there's a lookup function now I'm going to not say much here about this but we're going to look at the analog code written in go so if you squint I'm gonna go back again if you squint at the Oberon - code just about right it almost looks the same go back for it one more time the difference is quite obvious just in the key word so here we have different key words they're they're lowercase packing a module is called package but otherwise it's the same structure we have types we have we don't have a struct in oberon we have a record but we have also a function or you know in this case procedure declarations and look at that receiver type declaration here it's exactly the same so the observation here is that we really have the same the same structure here except that instead of the Oberon two tokens we have essentially C tokens but we have Oberon two structure we have the same concepts packages imports times functions methods etc and go we went a little bit further and we distilled some of the features even more so for instance there is only one loop constructing go the for statement while in Oberon two days there's three I believe so the heritage of of go is at least as much Oberon - or you know the Pascal line as it is C and in fact all these concepts that I mentioned here packages imports strict memory safety garbage collection dynamic type checks etc they're actually coming from Oberon - but programming language design didn't stop here object orientation and generics was becoming quite popular around 1990 and object orientation in particular was the paradigm that had established itself by now as a very valuable paradigm and concept in a imperative programming language and so we wanted to make sure that go can support it well at the same time we see also a proliferation of banana dynamically typed interpreted programming languages these languages often trivially support object-oriented programming by the way they are implemented and also because there is no static type system it makes it easy to write generic code in those languages such as a minimum function for instance that you know works as long as you provide the right matching arguments but maybe 10-15 years later we see a backlash it turns out that complex our code is really hard to read and some people have compared it as the modern - you know the spaghetti code of the 1970s and also a realization started to people started to realize that large programs in dynamically typed languages are really hard to maintain if the program becomes more than say ten thousand lines of code it's it's very difficult to understand what's going on except for the original author together with the object orientation and the fancy type system type systems languages also introduced a lot more notation and Rob Pike talked about this in his talk public still avoid a loss come 2010 still it was clear that we wanted to get some support for object orientation and the inspiration Ingo came this case from small talk it's always good to go back to the originators of something well similar was a little bit before but a little bit of a different nature so in small talk everything is an object and I can send a message which is really a method call to any object and if the object understands the message then you know everything is fine if the object doesn't understand the message then I will get an error basically an error that says you know I think I don't understand this message so in go we wanted to have somehow the same power but without the possibility of getting this runtime error of you know message not understood so that means we needed a way in which we could describe which messages were understood by an object and this led directly to the notion of interfaces as it's quite obvious but in hindsight but furthermore to really get to the same power as small talk we needed to be able to send messages to any object or in go terminology to any instance of any type typically that's not possible in an object-oriented language typically you can only send messages or actually call methods on instances of classes because in the implementation you need have extra information when you do a virtual dispatch and that extra information is typically an alert or so and that's usually somewhere in in that data structure which is usually you know a class or struct C++ so what we realized that this extra piece of information is actually only needed when you do a dynamic dispatch you can have methods and we have them in go without dynamic dispatch and they're completely free they are essentially just syntactic sugar so this realization led to the crucial insight that we only need this information when we want to do the dynamic dispatch which is when we call a method through an interface and so the trick is of course we store this information in the interface and not with the with the data so with this mechanism it became possible to attach methods to any type without having instances of the type having to carry around extra type information this is really the only additional machinery that was needed to make object orientation possible and go there's of course also concurrency support and I'm not gonna go talk about this much Rob has elaborated on this much more eloquently than I can also be G yesterday mentioned concurrency but I I'm gonna mention these two papers here if you want to learn more about the origins so finally we have generics or sometimes called parametric polymorphism at Eastham or which is the more precise term and it's probably the single biggest feature that's absent in go but what exactly is it it's most often missed by newcomers to go haven't really written a lot of code yet it is true it would be useful sometimes but we rarely miss it in fact it's also more of a type system mechanism it's not clear if it's an essential language mechanism so with all the features that we try to including to go we try to make sure these were mechanisms that had established themselves over a long period of time were found in many programming languages in some form or another and really carry their weight generosity is a very very complex subject it's both in semantics and in implementation there are significant trade-offs so to mention a few should a generic feature lead to duplication of code and thus large a binary or should it just be there for ease of expression but then potentially leave to lead to a slower binary or should we just replicate code in form of like a macro and then lead to larger source code so there's just a few things to consider it's also a non orthogonal feature it interacts with pretty much any other language feature so it's not that we haven't thought about it we even have prototype did in fact Ian Lance Taylor had several proposals and even implemented parts of them but we are not at the point where we feel like we really understand where enough so at the point we basically are holding off and keep thinking about it so how did we get together how did we get all this together into this nice compact piece this nice compact language called go so it helped for sure that we had the luxury to sit down for almost two years pretty much undisturbed and hammer out the basics what was crucial when we did this is that we added one feature at a time we pretty much would come in one morning and say okay we're going to discuss maps and then we would just work on maps for a week two weeks maybe longer until we had something that we felt like we would work and then perceived its repeat sometimes we would go back and change things but by doing this design in this way we made sure that every time we added a feature we had the mechanism to basically check it against everything else that we already had and make sure that didn't interfere in some ways and as a result we achieved mostly orthogonal design however this was a very hard process initially we were just a team of three people our bid very very different people and so the discussions were pretty intense sometimes emotional and it was a very humbling experience but it was really useful to have all three people in this room because everybody brought a different perspective on things and I think and Rob Mike mentioned that too I think we all agree that the result is much better than if any one of us had come up with something a little bit later Ross Cox joined the team and he really cut through the crowd pretty much he asked the right questions and he clarified a lot of things and he didn't just make it work he really made it work well so thank you very much for that Ian lens Taylor one day showed up and said hey I have a go compiler we we didn't know he was working on this and he in fact had a prototype running it's called GCC and now it's you know a real compiler it's GCC go and we were a little bit surprised but we were really happy to see a second implementation and in fact he proved tremendously valuable because now we had two independent implementations they were not sharing any code they were basically based on the same spec or at least we thought on the same spec turns out that different people read things differently and so the implementations varied in minor ways which was extremely elusive because we would find a lot of box in both the compilers and also in the spec and now we even have a third the front-end if you will the go types package which is going to go into the standard Reaper for 1:5 and it provided even a third validation and during the writing of code types we found box in the compilers both of them in go types and in the spec so this was extremely useful so we are pretty confident now that we have a much much better suspect than what we had you know a few years back so the language went through quite a bit of change over time so the the original design is quite different from what we have now and it was very useful to at the same time not just work on a language but also work on the library because it allowed us to vet our features and not design into the blue we also had this tool go thumped and then later go fix which helped us adjust existing code so when we decided that we needed to change something it was very easy to do because if the syntax changed we would just run go funked reading the old syntax and producing the new syntax if we wanted to change library api's we would rather go fix packaged sorry module and and and run it over the library so this allowed us to actually move library and language forward while he was still evolving this is something that was extremely powerful and crucial I think because if you don't have this mechanism you're at some point stuck with what you have you have too much code that you can't go and change so here's a list of features that came in much later most notably the optional semicolons the first first release I believe still had two semicolons and so there was a large amount of code out there not just in our code base but also outside and having go from fix it all up was tremendous we're still doing a few backward compatible changes from time to time the occasional bug fix but this is proceeding now at the very very lower rate so I like to say a few words about how I see the future of go it if I read the signs right it looks like the future is pretty bright but it's very difficult to come up you know with a very educated guests but we can look at some of the aspects of a programming language to figure out what makes it successful and so here is a somewhat incomplete list of things that help making language successful it's good to have a clear target we were really targeting systems programming and it didn't actually matter that it was the wrong target the first people that showed up using over Python programmers but at least we had a target and we were focused the most famous example probably is PL one a programming language that was designed to solve all programming problems didn't go very far but equally important as the language is the libraries and the tools it's completely useless if you have a great language but no libraries or no tools but the market needs also be ready if you have the language of the future and nobody understands its features and the market doesn't need its mechanisms you're designing into a boiled it can help if your language has some technological breakthrough for instance Fortran was the first language that implemented decent register optimizations and allocations for expressions alcohol 60 introduced vegetable program structure locality of scope recursion things like that model a 2 and ADA they introduced the notion of packages it's also helpful if you have a language without competitors feature with features without competitors for instance list was the first language that really had recursion it had a lot of firsts and they're basic at the time was much better than any batch oriented system and small talk was have had a lot of firsts in numerous aspects it was interactive it was graphical it was object oriented was extremely simple marketing rarely helps though Sun Microsystems bet the farm on Java and in some way they succeeded Java survived but son is dead so how about go so we clearly had a clear target behind the design it's a multi-paradigm language and syntactically lightweight so it appeals to a large group of programmers it has a few language features without competition at least for now Gouri teams and interfaces and defer until a few months ago swift now has a defer there are some tools without competition again until recently we have a fast compiler okay Pascal turbo Pascal had a very fast compiler immediately go fund it turns out that now a lot of other programming languages try to you know implement their own standard formatting tool go build a build tool that allows you to build your software without referring to an extra description such as a make file go as strong standard libraries and it has a pretty solid implementation that's only going to get better and we have pretty good documentation but really almost no corporate marketing actually no corporate marketing so we'll go ever become mainstream hard to say ok we need to cross the chasm from the early adopters to the early mainstream and I don't know if you're there yet it's a little bit like trying to figure out if you're at the end of a recession you know when the end of the recession is declared you've already been in recovery for a long time so but I think it's really important that the NGO community as a whole is unifying behind this goal we can't afford too many you know grave mistakes going forward and you know a grave mistake for instance would be fragmentation of the platform but we also need to keep in mind that it usually takes about ten years for a programming language to get established that's that's just the time it takes for a significant amount of people to really start programming in it and and for for people to really understand the language it turns out that even now seven years after its inception we still find new ways to write a particular piece of code and you know I cannot even conceive how many you know hundreds of years ago take for C++ get a same point so what are some pitfalls the language is really frozen but in some form some of these special comments build tags special interpretation of import paths internal packages rendering descriptions are kind of a form of language design for instance internal packages is a poor-man's mechanism to build your components so that people cannot access the internals vendor in descriptions is something where we almost went the wrong way so we need to be very careful of these things they're not specified in a spec but when you actually write a piece of code you really need to know and that here - the specification of these extra notational extra notations in your code so let me let me close with an observation and don't don't read too much into this I just sort of happen to walk into this observation in the 1960s language experts from both America and Europe actually teamed up to create alcohol 60 and these were all people that actually had experience in language implementation this is quite a bit different from designed by committee nowadays where there's often a lot of people in the committee that have never implemented a programming language in the 1970s the alcohol tree split up in basically two branches to see branch and the Pascal branch and of course both of these branches produced an enormous number of successor languages but in some ways it split up in the American NT and two European branch and somebody said at some point that you know a programming language reflects the personalities of the designers and as such it probably also reflects the countries of the creators and I remember an anecdote that somebody told me about Niklaus weird where somebody told him that he thought the language Pascal really reflects the state of Switzerland it's kind of small clean compact but somewhat limited in scope so now it's about 40 years later and in many ways these two branches actually join again and go we have both influences from C and we have influences from Pascal and if you look at the actual people involved and here is where I would say don't read too much into it you know we had originally three team members one from America and vom from Europe and then we had a third one from Canada who was a neutral arbiter so let's see if Co can enjoy an equally long run as its predecessors and thank you very much for listening

Channel: Gopher Academy

Views: 40,312

Keywords: gophercon, programming, software development, golang

Id: 0ReKdcpNyQg

Length: 41min 34sec (2494 seconds)

Published: Tue Jul 28 2015