A History of Clojure by Rich Hickey with Q&A

hi i'm richieke and i'm the author of closure and i'm quite thrilled and honored to have closure be part of hopple and sorry we can't all be together but i appreciate you coming and watching this talk uh so uh it's a long paper and a long timeline and i'm not gonna go through it all linearly but uh if you're not aware closure was developed starting in 2005 in a self-funded sabbatical and during that sabbatical i didn't know i was going to write closure or anything else but i was intent to do something that i found interesting and not care what anybody else thought about it and that ended up becoming closure but it's certainly the case that i wasn't trying to satisfy any external objectives and so there's a lot of opinionated aspects to the design and implementation so beware um hopefully closure is pointed at some problems i think some of these problems were things that i was aware of and some of these things uh were undercurrents if you would uh but the dominant one uh was in my opinion programming had become too complex i was a c plus plus programmer then a java and c sharp programmer and what i frequently saw was projects you know come up to speed build out get to a certain point and then get crushed by their own complexity and so having done that for a little over a decade and a half i was pretty tired of it had discovered common lisp and sort of fell in love with how different it was from what i had been doing uh and how much simpler it was and i i said that's what i want to do i want to try to figure out a way that i could program and lisp um but the other thing i fully expected was after my sabbatical i would be going back to being a professional developer so if i was going to make something i'd have to make something that could be used at work in consulting jobs and things like that so to break these down a bit uh program complexity i found in practice was driven substantially by the fact that programs were imperative and that state management became something that was not manageable after a certain point and even before you got into concurrency and some of the promises of multi-core at the time people were just drowning in the update in place programming that was common using object oriented languages in addition the languages themselves were just getting more and more complex especially c plus uh they you know it just had more features and i was quite enthusiastic about them initially but in practice applying them with best practices i didn't find that we had commensurate returns to the complexity and people were no longer pointed at the complexity of the problems we were trying to solve but pointed at the complexity of the tools we were trying to use the other thing that was my experience was that using object orientation as uh was practiced uh at the time in those languages so c plus plus java and c sharp uh you ended up with a ton of concretions over time uh you know classes are the way you do everything and you end up with a class for everything including things like information like customer classes and things like that and in the end what happens is every class has its own mini language and all consuming code for that class is specific to it and over time you end up with hundreds or thousands of these classes and a very brittle application with code that's very dependent on your decisions about how to organize things and how to name things and what you put in each class so if you're trying to make something to be you know something going to use at work the things i thought were cross-cutting that really mattered were that you can't actually just throw away your existing code base that you know i'm trying to imagine going in back to places where i was consulting or talk to my friends and say uh here's a new thing just stop doing what you're doing and do a new thing people have big investment in what they're already doing and can't throw it away another thing that's quite important to professional developers is the access to libraries in two respects one is you don't want to write everything yourself you don't have time to do it people are already then we're used to being able to find libraries for database access and io and graphics and um if libraries were specific to each language any new language you know you would be sitting there waiting for the language to re-implement the wheel or wheels uh so that's also a challenge the other thing i found having moved uh from you know the traditional commercial object-oriented languages of cfos plus and java and c-sharp to common lisp was uh you know there was this big emphasis on lists but in my experience linked lists in professional code bases were pretty rare they were rarely used everybody uses vectors arrays and hash tables for performance reasons and what i saw in lisp and other functional languages was you know the elegance and beauty of writing code with lists uh and recursively but um i knew that wouldn't be acceptable from a performance standpoint uh so these were things i thought were concerns like i said having pretty much not been aware of common lisp earlier in my career moving to it and discovering it later was like an aha moment i felt like i had been wasting my time becoming expert in in c plus uh and i built a bunch of things including things for for customer engagements where uh i prototyped or even built solutions in common lisp only to have them say that's great that looks really good please rewrite it in c plus or java or c sharp and the trend certainly at the time was towards java or c-sharp if someone was going to move away from c plus it would be to java or c-sharp and the whole idea of a completely alien language with a different deployment strategy uh was just not acceptable and then there was the whole lack of familiarity and kind of uh the knee-jerk response which is well known to lisps syntax and the parentheses and uh and whatnot the other thing i found in programming with common lisp is that they're uh it's not inherently a functional language but there's plenty of functional algorithms and functional use of it by convention there are a number of things that are functional in the standard libraries the functional things tend to have linear performance because they're built on top of the lists so things like a-lists or associative lists uh are quite easy to use but they have terrible you know performance because they're linear and the performant thing so common lisp is a practical systems language that has vectors and arrays and and hash tables were not functional they were imperative so not only did you have to switch away from functional programming but you really had to change your code the interfaces were totally different and so you couldn't write something small that you then turned into something large without really changing gears and rewriting other aspects of common lisp as later i'll talk about wanting to pursue uh functional programming was that the bottom uh of lisps were con cells which are mutable in addition the libraries that manipulated them manipulated that you know concrete data type uh not via some sort of an interface uh or protocol um so that's not great especially coming from object-oriented programming or at least interfaces or virtual functions are quite common to give you some indirection between the abstraction and the representation you'll see later i'll talk about some of my attempts to bridge between common list spend and java or c-sharp and they were not successful in the end the things i didn't care about and still don't uh are being novel inventing a new thing you know solving some research problem or whatever this is a tool building exercise designed to make a practical thing you could use for practical work and i know didn't really care and don't care if anything about it is new and i do not claim any aspect of closure is novel another thing i don't care about and didn't was whether or not it was purely functional trying to solve the problems of you know can you do programming purely functionally and and how to do that which some languages try to tackle um even though i just mentioned it as being a part of the objections to common lisp uh having worked with common list by who's firmly in the camp of the parentheses and the code is data and uh that notation uh is a feature it's not a bug and it's something you should deal with and i wasn't going to try to accommodate people who wanted something more familiar i was happy with it i thought it was a feature i still do and so that's not a problem uh again because commonly scheme both had these issues at the bottom i wasn't going to try to either embed in one of those languages or remain compatible with them so that was not an issue and i wasn't interested in any more of a type system or even as much of a type system as common list pad i like the dynamic typing and i knew that being hosted on one of these runtimes i would get at least enough of a runtime type checking system that i wouldn't want to be inventing something on top of that um so again closure is not new and in fact what was great is it you know it's not really a from scratch language design it's lisp it takes advantage of almost at the time i started 50 years of prior lisp knowledge and tons of papers and absolutely standing on the shoulders of people standing on the shoulders of other people so there's that the other things i did before 2005 were some experiments to try to get common lisp to work with these environments i knew i needed to you know have in production the clr or the jvm i had written a little toy lisp interpreter on the clr which was mostly about trying to interoperate to write in lists but interoperate with libraries uh another project following that was something called j fly in which i embedded the jvm inside common lisp using farm function interface and jni and that was okay although uh the jvm ended up sort of destabilizing the common lisp runtime which was more oriented towards running for uh long term within an interactive environment i mean jvm is great at running long term but uh the interaction and probably some bugs in the implementation made it uh less stable than common lisp on its own and finally in order to get a separation between the the two runtimes uh in process i made file which was again kind of the same interface uh in terms of interop syntax which worked over wire protocol so that you now have two processes one is the common list process and the other is the jvm or the clr the problem there was you know it didn't really do anything for the i really don't want common lisp code or a common list deployment you have these two things to deploy now but this work definitely let me play around with some of the interoperability syntax writing access to say java stuff from lisp and a closure programmer would look at this this is not closure and recognize a lot of it the calls to new the constructor calls with dot the dot calls for methods the ability to define interfaces in this case using this proxy and implement interface interfaces and call methods all those things were important and remain important are things that closure can do and the syntax was really being worked on already before i even decided to write a list or do a functional lisp so the thing i eventually decided to do during my sabbatical was make make a lisp from scratch as opposed to modify or embed uh another set of sort of cross-cutting concerns which had nothing to do with my experience with uh common list but had more to do with some of the work i was doing in c sharp at the time which was uh the election projection system uh was a a big you know uh uh statistical specification for statistical models that had to run on election night it was written in c sharp and we decided uh to write it in a functional style just sort of as a self-defense thing and and we did and we ended up with a few things i thought were interesting one was it was very much not idiomatic code other developers that weren't part of our thinking about how to approach things functionally would look at it and sit and just be bewildered uh by for instance having to track return values because they're used to methods that mutate things uh and additionally without any kind of persistent data structural library we ended up using you know the copy on right collection capabilities that were there which are not a great fit and not great performance so it really left me with uh functional programming is important it definitely reduced the complexity of that system and made it more robust but uh not being idiomatic and not having support for it made it hard and hard to uh get people to follow um the other thing i wanted to do was i decided to do was to host the language i was writing on the jvm or clr in other words to have it be a language that emitted by code and ran on those runtimes um and it to me hosting meant more than just the jvm or the clr was an implementation detail but that the jvm and the clr were uh accessible and that the language closure in this case would be accessible to it in other words it's not a layering thing it's a it's a it's an embedding if you will and that the interop would be bi-directional the other thing that had to be the case was it had to be fast enough to work where java or c sharp would not necessarily as fast but fast enough that i could uh sell it as a way to tackle problems that we would tackle in java or c sharp the last thing i want to do is to keep it simple and small i mean the big refreshing thing for me as much as kamalis is a very large thing most of its size has to do with a rich library the language is small not a small scheme obviously but the core of it and going all the way back to mccarthy you know it was sort of like what's the minimum amount of stuff you need to bootstrap a language and sort of following that you know what's the minimum language you need to be an effective developer and i thought it was a lot less than java or c-sharp and certainly less than c plus plus so keeping it small was an objective and remains an objective so uh then it was a sort of an implementation uh project you know what what did i need that i didn't have and the critical thing was i didn't have any high performance immutable data structures and we're talking about those hash tables and vectors i wanted to be able to have some alternative for state management to locks and and mutexes and that kind of thing um i wanted to have some sort of runtime polymorphism system so what i'm talking about here specifically is the kinds of things that are yielded by java interfaces c plus uh you know virtual functions um the ability for a program to interact with new algorithms for the same interface at runtime depending on the kind of data or the values of data that i was encountering i wanted to fix that aspect of lisp of it being built on abstractions i mean certainly understandable given its history but i wanted to redo that try to put abstractions at the bottom and of course the hosting so to do the persistent hash maps or to do the hash tables i ended up eventually finding bagwell's hash array map trees and all i did was make a ver which were not functional or or immutable what i did was just make a version of them that were persistent and immutable and just used sort of the ordinary path copying strategy it ends up that java's system array copy is quite efficient and made these competitively fast with java util hash map similarly the same kind of 32-way branching tree structure uh was a viable way to do vectors that were you know performed well enough to use so this is a critical thing uh if i wasn't able to pull this off i don't think i would have continued uh during my sabbatical with with closure this was essential i thought to having something that was was viable for doing state basically i was looking at ml and sml at the time and i saw they had the refs and i liked that you know everything was immutable except this one thing uh just a pointer-like thing uh but they weren't as far as i could see no concurrency semantics around the reps so the idea was just to add them in addition i wanted to get away from sort of read modify write approach that you would take with variables and instead use a commutation function say you know there's something already there here's a function applied to what's already there plus some arguments and get the new value that will be there so i i also took on writing an stm at the time because that was the only thing that would sort of be competitive with the multi-update possibilities of mutexes and locks and there's details in the paper it's a mvcc approach that doesn't need re-tracking uh it ends up that this was super important for the storytelling of closure when i first did that but in practice i think the stm is used extremely rarely there are a couple of other flavors of these refs in enclosure in particular atoms that just use cas they don't support coordinated updates but they are enough for most applications uh working on lisp the biggest thing was to sort of move away from the console which was the concrete data type to an abstraction over it which i'll call seek or i seek and because you'll see later i didn't actually end up with the with an answer to runtime polymorphism in time i ended up implementing the abstractions in using java interfaces uh and so there's an abstraction for con cells uh that support you know first and rest and and uh and termination uh but support an open set of implementations as opposed to being concrete uh there were a bunch of just decisions you make when you want to write a lisp you know is it a list one or a lisp two i liked list one and schemes being a lisp one but then you have the problems of how do you deal with uh macros and lisp two was sort of partially pointed at that problem uh i ended up approaching that by using a non-interning reader and this is probably a lot more less specific than a non-lisper would understand so i'm not going to go into more of it now the biggest thing though that i think is apparent to people who look at closure versus the other lisps is that there are more things and i think that they tend to presume that now there's syntax on that you know your broke lisp which was always about uh being data structures and i don't think that's the case i think the way to think about closures syntax is that the reader has been enhanced to support more data structures but exactly the same strategy as before the reader could read lists before now the reader can read lists and vectors and maps and sets it ends up the vectors of maps and sets are used very rarely for language syntax vectors are used a little bit but they're used quite a lot just to represent data literals but it's still a read eval system and it's still evaling code as data there are a bunch of sophisticated aspects of both scheme and common list that i just left out like continuations and conditions other things that i couldn't make happen that i would have liked to have like tail call elimination for hosting an interop uh the strategy was to dynamically compile to bytecode from there the runtimes uh can get you to native code so there's native bytecode and then native code and things like hotspot will get you from byte code to to native code uh so it's really great and and the jvm is a fantastic language target in that you you just have to go that far and then there's a bunch of quite mature infrastructure that can get you to high performance and inlining and things like that there was a little syntactic sugar that's kind of non-lisp in the dot operator that's not first class the way almost everything else is and then the sort of the big approach to hosting an interop part of this being hosted was to have some bi-directional conversion and copy-free compatibility by that i mean if you have a java uh thing you know a java arraylist or java hash table and certain closure abstractions make sense for that they just work you don't need to copy your java stuff into closure data structures similarly if there are java abstractions like iterability or being a collection the closure implementations support those abstractions and those interfaces so if somebody has written something in java and expects a collection to be passed closure collections are java collections and there's they satisfy those interfaces and they do not need to be copied or converted um and where possible if the semantics match i would try to use the java thing directly so strings were immutable in java that's good enough for closure and so closure strings are java strings uh it was a a common question i think of uh the reviewers and whatnot you know sort of like how did this work from a financing or funding standpoint uh so i'll talk a little bit about that now like i said i i decided up front to take two years off so i sort of had that framing uh and take some some of my retirement savings to do that and i did take that time to do the core design and the bulk of the implementation including the data structure stuff i just talked about and the syntax and hosting and by code generation all in this time frame i had originally intended to target both the clr and the jvm but i focused on the jvm for a bunch of reasons one is i think it had stronger open source and library support and the other is i thought it had better dynamic support than the clr which tended to be more focused on compiling up front um in the same time frame i met and married my wife who married me in spite of me having what seems to be objectively a terrible plan to make an unappealing language for no money with no prospects of making money on it and steph has been a huge support throughout the entire time of development and then by the fall of 2007 i was ready to go to the local lisp meetup in new york city and present yet another lisp uh having done that uh for some reason uh there was a reasonable amount of radiation of the existence of uh of closure on say uh hacker news and things like that and i got a lot of interest and some early adopters and i set up an irc channel and a mailing list and sort of lived on that and saw people show up and asked questions and uh had very high engagement uh with the early uses of the language when i initially released it there was a very small core library and so the next year or two were spent um enhancing the core library adding some language stuff but primarily rounding out the library making it make more sense making things complete improving the documentation but people could show up on rc and be like i have your map and i can't do this with it and in 15 minutes i might push a new version of closure that gave them some new capability uh my friend eric thorson with whom i worked on the election system also gave me some consulting work and bits through here which helped supplement uh the retirement inc you know fund but this was now two more years beyond what i anticipated and so eventually by 2010 by the end of 2009 i knew this was going to come to a dead end from a funding standpoint i had a funding drive with the community and had several companies and some individual users participate in funding ongoing development that yielded partial income i only did that for the one year because uh in the end when you ask for funding you end up getting more bosses than you want and the number of bosses i wanted was zero uh but it just kept going the language grew the number of users grew the adoption by companies grew and uh i just was fully engaged in in building closure and enhancing it and supporting it uh but critically in 2010 uh stu holloway who wrote the first book on closure uh and justin gatlin we're running a consultancy and doing some closure work but keen to do more and uh i went down to meet with them to sort of figure out how we could work together and pitch them the idea of daytonic which was a functional database concept i had been thinking about and eventually partnered with them to build that stu worked with me in building it and justin ran the consulting company that supported it uh in 2013 we merged the the daytonic effort with the broader consulting org which was called relevance and we formed cognitect which was at this point now a consultancy that was substantially engaged ins in projects that involved closure and atomic and doing this gave us revenue uh and that allowed us to support the community more allowed us to hire alex miller who's a critical key contributor to to closure and also uh sort of the prime interface between the development effort and the community at large and a sweet very nice guy and we're still going so cognitect ran sort of the sponsorship if you will of the closure development effort by keeping me alive and uh and cognitech grew and daytonic grew and to be a successful product eventually last year uh cognitive and atomic both became part of new bank which was a big consumer of daytonic and i'll talk a little bit more about that later and that's been great also that's more staff now working on both closure and atomic in addition at new bank we're able to directly fund some community members who are working on libraries and and training and things that we think are are useful to the community speaking of the community the community is great again there's a lot in the paper about uh about community and sort of historical details but uh it's obviously one person can't support everyone in the community has been great at supporting new users they've written books they've given conference talks and built libraries which is important being a lisp it's a small language and most of the growth should come in libraries not in language enhancements in addition we've accepted bug reports and patches from the community but closure never really became sort of a collaborative community development effort i personally didn't anticipate even people expecting that in open sourcing it and it was probably my naivete around how open source had been working and what people's expectations are of open source meaning open and collaborative development and i would say closure is somewhat atypical for open source and being uh open but not collaborative uh and i think there are some advantages to that both personally and in in the result uh but one community thing i want to call out is uh during the the course of closure i i started an endeavor in 2011 to make a version of closure that could run on uh the javascript runtime and particularly in browsers and i built the initial version of closure script in 2011 but eventually david nolan took over the stewardship of that and as you'll see later i think followed along with the stewardship model of closure itself and has done a terrific job in closure script is much more of a community effort and not something that i've driven other than launching it so that's a big community success story just so you understand a little bit about what kind of people come to closure and use closure it's not lispers it's not people who used common lists for scheme before it's overwhelmingly as you see here uh object-oriented programmers who adopt closure java is the number one place people come from but you also see python javascript and ruby and and cnc plus which makes sense to me um another characteristic of closure programmers is they they trend more towards the experience side but by no means this is uniform so this diagram plots years of programming experience and salary for a bunch of programming languages and you see closure way over to the right in years of experience so this graph starts at seven and a half years so it's not like linear uh back all the way to zero years uh up there with f sharp and erlang so we're a bunch of languages for experienced developers who are tired of it and are looking for something a little bit uh more rigorous being functional uh there's a bunch of stuff in the timeline here things that you know features you can just see that there's been a regular release cadence the feature development is front weighted but people have written books i have not written a book on closure but many people have libraries arose around especially initially web programming but also logic programming and database programming and things like that in addition i'll talk later about talks but some of the big uh motivational and influential talks that i gave were done in this time period moving forward again more releases some more features a bunch more in the tooling space and a regular set of closure community events and conferences so there's now several conferences held every year at different places with different sponsors around closure and at the time i started writing the paper for hopple which is now a couple years ago there were already 32 000 closure github repositories and 20 books and there were closure modes for all the editors and things like that here are some of the books and you can see that they range from books about sort of the ideas behind closure to just how to get things done to how to do very particular things like machine learning or high performance computing or web or things like that a bunch of tools have been built this is a one example which is a closure environment for intellij which is popular java editing environment and the reason why i like showing it is because it shows what it would to a java program would be a very uh recognizable development experience with structure navigation and project management syntax highlighting code completion and embedded documentation and things like that none of that is precluded by closure being a lisp or closure being dynamic so this is the kind of thing that the community has built in terms of tooling that doesn't count at all some of the tooling we get for free by merely being implemented on the jvm like debuggers or profilers and things like that that just were written for java but work with closure by virtue of the fact that it emits spike code it's to sort of editorialize a little bit in retrospect i think things you know i'm presuming a lot of the audience here are language designers looking to get their languages adopted and so if you want to try this at home some of the things i think matter to closure are being very accessible so like i said i was living on the irc if you read the logs back from when i first released closure sort of any time of day i was on there i was fascinated that anyone was using it so it was interesting to me to see people show up and ask questions but i i did and it was addicting to be there and and quite thrilling the early days of of closure on the irc also the mailing list i think having a unique name for closure is important i strongly recommend you come up with a unique name for your language so that the search is good if you overlap with an ordinary uh word it's just going to be harder to find your stuff my brother made a great logo i'm sure that mattered i gave a lot of talks i gave talks at developer conferences this is for professional developers mostly not about closure primarily those talks were about the ideas behind functional programming and were very general but i also you know trotted out to a bunch of user groups the java user group in new york city and also uh some some hardcore lispers at mit to give talks about closure you know closure for java programmers closure for list programmers and uh and all these talks event eventually ended up on youtube and i think that that is huge uh having talks talks of aimed at specific audiences about topics that matter on youtube is important uh closure being hosted make made it easy for people to sneak it in it wasn't like we're going to adopt closure today you know the typical java application is using hundreds of libraries and if closure is one of them i don't know how anybody really knows and and it absolutely has gotten adopted via the back door that way uh i do think that the problems i enumerated earlier were real and and that people share those problems they don't often share the realization that they have those problems and i often hear after someone has adopted closure i didn't realize how much stress i was suffering from writing code in a in a an imperative way but they are real pro you know problems of programming i think that closure isn't a gentle intro to functional programming because it really only takes on uh idiomatic immutability and and programming with pure functions and those data structures and data literals it doesn't you know force you to also take on sort of a bigger a bigger uh thing in an elaborate type system uh and uh as far as closure script the ability to run the same language in the browser is quite big for that there are lots of closure shops that that write both the front end and the back end of the code enclosure and and closure script quite importantly is closure it's not a subset it's not a dialect um it's it's pretty much fully closure uh i think closure you know in the end solve some problems that this is definitely monday morning you know retrospective quarterbacking i didn't set out to solve but i think uh it ended up being suitable for solving one is how do you do information programming i don't really have time now to talk much about that but i think that uh that there's a big difference between programming the mechanics of a program uh you know the data structures the compiler and and algorithms and and and what kinds of constructs are useful for representing mechanisms versus what kinds of constructs are useful for representing information the stuff we happen to know about this customer the stuff i need to know about this customer in this context that's really that kind of programming dominates application code i talked a little bit about creeping concretion with classes before not having any classes means you don't have that problem similarly i think that when you have a object-oriented class-based ecosystem the the framework that comes with the language is dominant and it's hard to get traction on third-party interfaces and frameworks because everything enclosure is data and data out it's easy to decide how to approach the interface to your library and it's easy to have independently written library things that work together i i think that you know to the extent that systems are now largely composed of services independent processes that interact over wires using you know transporting fundamentally shared understandings of data structures across those wires working in a language that does the same thing all the time works with fundamental shared basic data structures all day long uh means there's a lot less impedance when you want to participate in a service uh based architecture your you don't have parochial views of how data is represented you have quite common views of how data is represented and you're ready to go on and off a wire i think that's that's big um the other thing that closure avoided substantially was was instabilities for people who were using closure and production uh because it wasn't ever uh intended to be a place for experimentation and the stewardship over time has very much uh issued experimentation in the language you can see that in this diagram this is a code retention diagram so it shows when code has been introduced colored by year and then what happened to that code over time so if the code got changed or removed you would see that that band would would diminish across time and you can see i think what i would call a nice layer cake enclosure in that there's a high degree of stability we definitely tend to avoid changing things except to integrate fixes but we don't change things just to change them or because we think we have a very slightly better way to do it in an effort to maximize the stability for people who are consuming it and that is overwhelmingly part of what people would report in using closure is their very old code works perfectly well and they're happy to update to a new version of closure because they know we're not going to mess with them and it's a credit to david nolan that they've run the closurescript ship in a very similar way layer cake so i'm going to end with an anecdote which is neither small nor short temporally so i think it's uh it may not be indicative of everyone but it shows what's possible a new bank was a startup in 2013 looking to build a digital bank and change banking in brazil and elsewhere and so start from scratch in 2013 through today where now there are 1800 engineers uh most of whom had no prior functional programming or closure experience that's important because i think people are reluctant to adopt closure or functional programming because you know it's different or whatnot but this proves you can certainly train people get them up to speed and have them become productive uh the overall system involves 577 closure services and 165 lambdas there are about 3.9 million lines of closure uh at new bank that is an unbelievable amount of closure because closure is pretty concise although if you divide it by the number of services you'll see that size of each service is actually competitively small with developing services and other languages there are 320 daytonic databases so new bank was the atomic customer is the atomic user and a big one so you're talking about applications written in closure running on a database written enclosure there are about a thousand deploys done per week 60 billion http requests handled and 30 billion kafka messages consumed and this is yield yielded a significant uh business as represented by the time final two facts here so i think closure you know does what i intended it's something you can use to build uh production programs uh and i couldn't be happier to see it used today by uh by so many people and thank you very much for attending this and i'll see you at the qa okay thank you rich for interesting talk and we have some questions queued up already so let's uh dig right into them i'm going to start with a question from brian harvey could you say more about the open but not collaborative development to what extent you accept or solicit poll requests from others i ask because snap is facing exactly this issue jens wants to do it all himself but people are clamoring for things he doesn't have time to do uh so actually we don't accept pull requests at all because pull requests are not values we do accept patches but not really in the area where people are looking to invent or enhance primarily we only look for bug fixes or documentation fixes and things like that so it is open in that sense there is a place to report problems and or submit patches with problems uh but it's not open in that uh it's not there's no place to sort of say i'd like closure to have this feature and and add the feature um i don't want uh my own feature enhancements nor others i i think the effort to keep closure small means uh i'd like to see the emphasis on libraries and so most community input is by libraries okay question from paul pfister what are the features that you've deliberately chosen not to add to closure feature wise and not back compatibility wise but you believe would be nice in another general purpose lisp for example features like first class environments first class sex spur first class continuations i guess the question is if you wouldn't put it in closure everything would be nice another lisp you know what's your what's your wish list uh so i i think uh you know to the extent it's an exercise in uh less is more in smaller languages or more i think most of the things that were listed there you'd really have to make an argument for why they pay for themselves from a complexity to benefit standpoint i mean it's hard to say to talk that way about continuations to a schemer but uh the only thing i really missed and it was something i would have liked actually but i couldn't make work would be tell calls that's certainly something i think all languages should have i'll follow up with that if the jvm supported tail calls would it be easier to put them into closure oh yeah if the jvm had telecalls closure would have telcos well if they had it when i started closure that would definitely be the case i understood okay a question from theodore uh how has the kind of problems closure is meant to solve change from 1.0 to 1.11 and then there's another follow-up question let's start with that i don't i don't think that the kinds of problems uh have have changed uh i don't think that closure has had sort of any kind of special purpose facilities from the beginning uh and uh it's been general purpose so it's used widely across you know uh medical applications and finance applications and web apps and data analysis so in that respect i don't think that the problems it's looking to solve have changed and it certainly hasn't taken on something you know uh specific to the current trends like differentiability or something like that okay the follow-up is a comment followed by a question clojure has long been a good choice for long-running processes though a repel workflow has advantages in other domains interactive programming is important in data science and scientific programming a tangible runtime makes it easier to benchmark high performance code data visualization from a reply is interactive are these happy side effects or design targets i mean i think their design targets of lisp before closure and enclosure is just you know agreeing uh i think interactive programming is is critically important and being able to try your ideas and see what's happening to have your context loaded while you're developing code and not have a sort of compile run cycle it just changes the way you think about programming and certainly agree about the data science you know applications as well i think it would be still it still remains a growth area for closure to sort of seek out some of the facilities that were common and lisps and small talks you know from their heyday we don't have rich environments like that yet and i think that it's fine to sort of meet people at the interactive environments they're used to in you know intellij and things like that but uh it's nowhere near as good as as the heyday of of of lisp and small talk i think there's a lot more tool work that could be done to bring us back to where we were will crichton of stanford asks many dynamically typed languages like python and javascript have been moving to ad type systems to manage complexity in large code bases is typed clojure the future or do you think there are aspects of dynamism that are helpful for reducing complexity that aren't accessible with static typing this is always going to be the big the big question so uh i i don't think that they're going to achieve what they intend with that i i don't think the types especially as applied to something like typescript or or java or c plus uh are rich enough for in solving significant problems especially if your program is still going to be imperative your problems will be dominated by state and not by the things those kinds of type systems can help i do think that there are costs and i sort of hinted at it when i talked about this uh creeping concretion uh when people apply types to information and end up with classes for customer and and whatnot they end up with far more specific code than they ought to and they end up with far less reusability of their algorithms and their libraries and so i think it's not just about dynamism i think it has to do with are you programming to data structures with a generic notion of accessors say for instance for associative data structures or are you trying to do information programming with objects or types and therefore sort of a full enumeration and i think full enumeration of you know a customer has x and y and z yields brittle programs that are hard to extend uh that have a lot of specificity are many times larger and far less general and flexible so it's a cost-benefit thing and i think that there are some benefits obviously to types but the costs are under-examined all right i have a question for you one of your slides referred to closure is providing a general introduction to functional programming and it listed as examples idiomatic immutability and pure functions i'm interested in what you would consider not so gentle aspects of functional programming for example is it higher order functions like map and reduce or is it something else that's less gentle uh you know so today i think people have more familiarity with map and filter and whatnot because it's been common in well mapreduce became a an algorithmic exercise for data processing outside of programming languages and also it's more common in other languages no the biggest thing is still the biggest hurdle is still immutability uh not having loops that update in place even local counters in place uh you know capturing return values and uh as values is still the hurdle and as gentle as closure could be it's it's not something i think there's a syntactic solution for if people have been developing in an imperative style for a long period of time there's just going to be a hurdle to having that taken away and just saying what do you do instead okay and a related question from conor huxtra of nvidia closure is one of the only languages to have symmetry in the naming of their reduce and scan algorithms reduce and reductions in closure is there a story behind the naming here or how that came to be uh well so we we have a thing in the closure community a meme especially from the early days about you know playing the naming game and uh i think we we definitely i love naming things i think it's one of the biggest challenges and uh we work really hard at trying to do that well uh in terms of reducing reductions yeah there are a number of things like that certainly when we got involved in the transducers that sort of force you to sort of lift out one more level uh as you take the source data structure out uh you end up needing to name that other thing but uh i don't think symmetry is sort of its own uh target for closure it's just sort of being careful about names okay uh here's a uh question from richard gabriel i have some information pipelines to brazil and i hear a lot about the atomic and not as much about clojure unless i mention the connection can you comment on how daytonic has interacted with or driven closure adoption at least in brazil and perhaps elsewhere uh yeah i i don't think i could talk about brazil specifically but it's certainly been the case that we have had people adopt closure because they adopted daytonic first so a lot of financial institutions they like the idea of datomic it has great auditability and tangible transactions and sort of you know how your database got to be the state it's in and you've lost nothing because it's it's append only or sort of accrete only of course they get there and they see the principles that underlie closure or present there and there's a absolutely uh minimal friction in using consuming daytonic from closure so uh yeah i think that many people adopted closure because they adopted daytona but not not the majority of closure users a significant portion of daytonic users chose the database first in the language second okay in a previous answer you referred briefly to transducers and i think that's actually a fairly central concept in the design of closure could you talk a little bit more about that and you know how that idea grew and how it's used well i mean i wish i thought of it early it was a late it was a late idea but it was born of you know looking at the challenges of closure has a laziness construct and you know when you see laziness anywhere in haskell or whatnot you end up with this whole uh how do you eliminate the overheads associated with composing streams and a lot of different ideas around that i had worked on reducers at one point to try to get more parallelism in play but the still sort of the raw overheads that for instance facilities like stream fusion target were present in closure and so transducers were an effort to sort of solve that problem in closure uh having sort of developed that uh were ready to go back that would be you know a foundational concept in closure i didn't think of it back when i started closure but where i would like rewrite closure transducers would be at or near the bottom of any kind of uh processing methodology because i think it's sort of a pure way to talk about algorithmic composition sort of independent of the source and it both works with you know pull and push sources and with data streams and event streams in ways that people have had to do parallel systems before okay so uh speaking of parallelism i'm i'm curious to know uh to what extent the concurrency features uh so for example you mentioned the refs and uh supporting either compare and swap or a read modify right kind of thing or uh in your case uh commutation functions that we just sort of provide a transformation on the value in place i'm curious to know to what extent those features are used by end application programmers or to what extent their users are mostly buried in support libraries which are then used uh i think they're mostly used by uh end user programmers and rarely by libraries because the that aspect of a program is usually at the edge so if you're gonna have an algorithmic you know library that does something interesting it's going to be almost 100 functional and then you know if that stuff's going to have to go over a wire or be the state of a program it's the application programmer who's going to choose to put a reference to that state in an atom so they end up being edge and therefore end user concerns and rarely in libraries if it was in a library that library would essentially be stateful and have kinds of you know objections to its use because you'd have to understand how the inside of it worked in a way that a functional library doesn't so i don't see it much in libraries at all it's left to the user to do that deal with the edges of their program i suppose the question i was really asking is is whether libraries of additional communications or parallelism abstractions get built out of the lower level ones and it sounds like the ones that already provided are doing the job yeah i mean like i said even the stm rarely gets used these adam and adam holding a pointer to uh uh associative you know persistent map is is the 95 case i think for applications and you know we've written large applications daytonic itself is an entire database and it has it has you know one atom around the database uh that's it and it uses some cues and whatnot for data flow but you just you end up needing very little okay a question for uh again from will crichton at stanford i've noticed that you use a number of terms like concretion and complex to talk about specific aspects of complexity in software design how did you decide on these terms do you feel there's a missing science of complexity management that we should be exploring with these kinds of concepts so uh you know a complex came from a talk again you know i love i love the dictionary and uh and uh it means something very specific in braiding and what i was trying to do is give more uh specificity there's an old notion which is coupling and i think that people write systems understand coupling sort of intuitively but if we want to uh sort of address problems in larger systems that involve extensive coupling i think we have to have more specific terms that i identify or call out what exactly where does coupling come from and uh so complexing is you know is a word that is very specific about a kind of coupling uh the concretion thing i think is uh again it's something i've started to use and it's a hard concept to get across to people when they you know because i don't think it's like many things you don't understand you have the problem while you're having it necessarily and so if you have a program and you have hundreds of classes that are each specific to a particular kind of information each class again i'm talking about ordinary object-oriented programming each class defines its own language and we've looked at java classes for instance for doing http and one typical java class will have four independent inside one class four independent implementations of what you would call an associative map with different names everything about that is bespoke and all the code that consumes or interacts with that class is also bespoke and so that code is concrete in such a way as to be specific and and also concrete in such a way as to be inflexible in other words it can't be applied to some other class and if one side changes the other side has to change or they'll break so i literally mean concrete in in in it's in the sort of rigidity uh but yes i don't know i i think there are so many conversations we should be having that were not about uh how to make software more flexible uh and also about programming over time all right a question again from connor hookstrove nvidia the thrush operators aka threadfirst threadless macros and how they play with algorithms enclosure core leads to some extremely beautiful and elegant code can you shed light on what went into the designing the algorithms so that they work so well with the thresh operators uh i so there's two aspects to the to those operators one is it's uh it was always an argument amongst list programmers that like of course you should know how to build things you know from the right side back out and have all these nested parens and a pile of closing right parens but i actually don't agree i think that that's something that needed to be fixed so there's just sort of the sugar of it specifically though there are the two flavors of the thrush operator one with a single arrow one with a double or say a single greater than sign and one with a double and that is an interesting difference and in there i think that there are two flavors of accessors and this is something i had to explain to people a lot early on one are more object oriented so you have a property of a thing in which case the thing will be the first argument of the function so to this to this thing do some stuff uh and that's the single arrow operator the other is when you do processes like mapping and filtering that involves streams you often have the data you have up front is not the target but the parameters so i know i want to filter this stream with this function or i want to map this function across the stream in which case the last thing you want to supply or so the thing you want to bind last is the collection and that yields the two flavors of thrush operator and really all you have to do is correctly assign your function to one of those two categories and then you get then the threshold bar seems to be incredibly elegant but it's really not there's only two ideas and i have both of them there and if you pick correctly it will it will look good okay i'll ask you one last question is there anything else you want us to take away about the design of clojure about your experience in uh building it and building user community i think i think the biggest thing is just that uh closure is really small and simple and it's adequate and i think we don't uh we're so ambitious in the kinds of things we take on in programming language design that we don't value enough that sort of minimal minimality and you know programs are engaged in very very complex domains now and we should be out of their way our our tools should be out of their way not complex things in and of themselves but if it's not required i think it should be taken out and we should try to allow programmers to solve problems with smaller solutions because i think that you can and something like new bank being built out of closure i think is evidence of that that's a giant system okay thank you rich i'll conclude by echoing a last comment from theodore hegeland i just want to add a thank you for keeping closure simple well you're welcome and thanks very much for having me uh be part of uploads thank you special q a okay thank you that ends this session

Channel: Visas Meilas

Views: 8,188

Rating: 4.9879518 out of 5

Keywords:

Id: nD-QHbRWcoM

Length: 67min 1sec (4021 seconds)

Published: Sun Jul 11 2021

Reddit Comments

[u/jobe_br]

I really enjoyed watching this when HOPL released it!