REBL - Stuart Halloway

good morning it is an absolute thrill to see so many faces some that we have all been seeing each other for a decade and some that are brand-new here it's great to see people who have come together to learn about closure to talk about closure to meet other people doing closure to collaborate together and to build great stuff it turns out that this side blank is not the title of my talk it's actually this slide also blank this is actually how features get developed in software like I don't like showing my slides before the talk begins so I put these two slides I'm at the beginning which is a very tactical reaction but eventually there should be some feature in keynote and things like it but it starts with the opinion right so actually the title of this talk is rebel and you've probably heard that being in closure we have this amazing thing called the repple and it is absolutely I mean forget about the technology the repple is a rhetorical stick with which you can play a drum right boy this thing is so cool and at Strange Loop earlier this year I gave a talk titled running with scissors where I showed a picture fairly similar to this one and this picture is about the flow when you're working at the repple so if you consider the entire flow with a human possibly at the front somebody can make text which can be read and then that becomes data or a program could just make data and then that data is passed to eval where other programs via the macro system can transform the data before it actually becomes program and then the result of evaluation becomes data which is then printed and then the virtuous circle of life continues so this is a great thing that being said it is common knowledge of anybody who's ever seen a rainbow that the purple end of the rainbow represents truth and justice and the yellow orange red end of the rainbow represents tribulation and so when you look at this slide you should look at that yellow image and say you know what it's a little bit disingenuous because the results of eval don't have to be data it can be anything right absolutely anything could come out of eval and that poses a significant challenge to the I have scissors for my data story because the scissors really come from you know starts with this perlis quote although rich has a variant of this quote but it's you know we live in a world where we have scissors because we have a powerful set of functions that can work on all of our data and we don't make the mistake of putting our data into walled gardens that require new api's and new functions to manipulate except we do that make make that mistake whenever we call eval because of course you know to be fair 95% of the time evals making data probably in most of your programs but that other 5% probably ends up representing an important part of the complexity of dealing with your domain and what is your poor program to do notice your poor program is now Gus got a bad end of the rainbow coloring as well and so when we reach this point when you're writing a program there are choices that people end up making and one of those choices is you know what we can try to print it somehow and read it back and maybe have data that our program can continue with or we can just have our program try to work directly with the thing that's not data in either of these cases we have often lost our scissors right we no longer have a common standard library for minute baiting things we now have bespoke libraries and often lots of them so that's a problem results are not always data I didn't even mention another aspect to this problem which is that results can often be large what happens in the printer when results are large we have an awesome wonderful flexible system of control that does exactly what you would want in every case no we have a really inflexible system that basically says you know I'm going to set dynamic variables for a print length and print level or probably one of your first 10 replicas periences you're not going to set those dynamic variables and you're going to stare into the son of an infinite sequence and then you're gonna have to shut down your repple and start over so your results are not always data and when they are data we don't always have good ways to look at them when they're large segue that's all really interesting I have a totally different practical problem right in my day job I work on day Tomic diatomic on-prem had a console and atomic cloud does not have a console I need to make a console for an atomic cloud that atomic team should do that people would really like having it and I have to say that while the console had pretty good leverage to effort involved in producing it with the on-prem console that kind of user interface is needlessly specific right we live in a world where we want to have scissors we're going to have these generic libraries it can many of a data and yet we end up writing these bespoke things I mean why is the day Tomic cloud console a special thing instead of an instance of or a variant of some kind of generic data console and this is a great sort of little micro story about how software design sometimes proceeds because you have a small individual personal problem right my problem the atomic console and you have a non gory going sort of thought about this yellow circle on a diagram and then eventually and it turns out if you line a hammock it helps with this eventually those two things sort of merge together and become something and so this talk is about that something and that something starts with tiny tiny additions in closure 1/10 there is a protocol called data Phi that says you know what we can get out of this game all right instead of saying that when we get back you know on the outside of eval that we have to have something that might not be data we can have a generic protocol that anybody can take their non data thing and that protocol and have a standard way of saying we're gonna get back to data now we have our scissors again hooray but when we do that we still have the challenge of navigation we still have the challenge of if the thing is enormous right or the thing represents part of something that I want to might want to find out more about later so there's a second new protocol in closure 110 called nav and data Phi is a generalization of how to get to data and nav is a generalization of how to get to more data and so now you can have a world where the data representation of something itself implements the nav protocol which represents a virtuous cycle where you can go around again so now you can say I got some result out of my eval and that result the data fication of it is a chunk of it that lets me look at it and in that data fication is the intelligence to get further and how can that intelligence be there I mean this is one of the places where an object oriented programming people would immediately say oh we need to make a wrapper or a facade or an implement that I can get to the next thing interface but another tiny little addition in closure 110 is the ability to attach protocols to data via metadata so now you can say I want to add this capability without having to make new things without having to make wrappers I can take a piece of data and say I want to navigate it this way of course that also means that by changing the metadata I can say I want to navigate that exact same data a different way in a different context when I'm working on a different problem and so when you put these things together data fie and nav and the ability to add protocols especially nav although you could data Phi as well adding protocols to data via metadata you have a generalized facility for laziness right seek is a specific facility for laziness it says that there's a sequential thing and there could be more of it this facility takes that to trees right this could be a tree thing it can be an arbitrary tree of stuff right I have some stuff I can probe down in it over here or I can probe down it and over there or I can not care I can go deep I can go wide I can go long the choice is up to me with these tiny little protocols but that's not very concrete so let's build something so I want you to imagine a web browser and the web browser is going to be a little bit different from your traditional web browser in that on the left hand side there's gonna be what you're looking at and when you click on a link in that web browser instead of going there there's gonna be a right pane that shows you where you can go next and so on the right pane you see where you have now selected and then via arrow controls you can say I want to slide through I want to make the thing that I looked at the thing in the right pane I want to make it the next thing and then I want to continue to navigate from there and in fact this is probably better if we just do it right so on the left hand side here I have a web page this is the wiki page those of you who are at board game night last night this is the wiki page describing the wiki and if I click on a link here there is a 50% chance given internet connectivity in the room that we're going to navigate on the right-hand pane dududududududu yeah really should I try another there we go all right so the internet is slow and now I can move that forward and make that the master the thing that I care about and continue to navigate I'm not going to keep clicking on the web and watching how slow the internet over my phone is because I'm gonna give you all this application at the end of this talk so then you can see how slow the internet is on your own devices but I want to ask the question what is so special about this navigation model of the web right there's nothing special about it at all what if this thing was not a web browser but a repple and what if I could go grab an arbitrary piece of data I'm not gonna make you watch me talk or watch me type and if I put some data in that repple and evaluate it and that data becomes a navigable thing that's not different from navigating on the Internet so on the left-hand side I have a list of all the stuff that I've done at the repple including by the way evaluating a java.net URL which is what turned it into a web browser but now I've chosen to evaluate something else right I've evaluated something that's a bunch of data and over on the right hand side I have a view that's a table view well why is it a table view it turns out that there's a very small number of data shapes right you this is you've drunk the closure kool-aid right there's lists there's maps there's you know whatever there's also a very small number of ways to visualize those shapes that number is a little bit bigger right so for a given shape you might think of different ways to visualize so you might go over here and click on the viewer and say you know what I'd like to view that as Eden right because I could see that data a different way or I might want to view it as a map now I can navigate into this data so I can go in and start looking at these individual pieces so there's a scalar data here is a collection of numeric values a different kind of collection of numeric values now of course with a collection of numeric values you might want to say I'd like to be able to view that as a chart right that's a general-purpose facility I don't need to have a special charting thing it's just generic data and of course with some of these things I want to try this one try some of the collections have more viewers so here I can view this as a collection where the values are just in one cell on the table or I can know that it's tuples and I can break apart that and see the tuples broken apart in the table again these are all entirely general-purpose facilities of course I can look at code I probably don't to look at code as data I'd like to look at that as code again but code is data right I could look at it and state it if I wanted to and we can configure this so one of the other things I haven't mentioned yet is that the top part on the right of this viewer shows the metadata of whatever piece of data this piece of data the numbers 1 2 3 in a vector also has the metadata rebel XY chart title that sets the chart title so you can extend the UI and customize the UI just by adding metadata on your data and say I would like to see this is what metadata is about right I'm going to do something different with data in a different context I'm not changing the identity of the data I'm not changing what it means I'm just saying I want to see something different so you could extend that via metadata so that's all kind of interesting but I haven't actually showed you those protocols in action yet right this is all just data right this is a fancy inspector at this point but what if we took some data like exceptions you could have a data five for exceptions that returned a specific view of exceptions that you could drill into go and look at the stack trace drill into the individual stack trace elements and so on and so forth let me get back out to the exception here someone get to the exception when we started working on this we had a actual custom viewer for exceptions that may in fact go away now that exceptions are data fide although maybe exceptions are enough of a special case to warrant you know continuing to have that kind of special viewer but let's back out again what about a spec well spec has a very simple help navigation and date if I help her in rebel might specs are not data fighting closure right now but it has a helper in Revell that will return the form of the spec so you can get back to the form and look at it expect to see I'll see in here a lot more about spec later in the cons but I don't want to I won't steal any of that thunder right now what about though if we had something that was recursive and arbitrarily large like a file system so this is a nav over java dot io files oh i have to turn on the nav so the nav in files is not actually shipping enclosure yet so that's just a code snippet so let me go run that let's see where did I put it and of course you can enhance this as you go did I put it you can enhance this as you go if you know where you left the code here we go so now let's try that again so now I've extended the UI by adding nav afiyah file systems now all of a sudden file systems are recursively navigable so if I hit a thing that's a Java dot IO dot derp it'll tell me the modified time and the name in the path and I can drill in and I can drill into files and having drilled into files I might hit an HTML file which I can see the metadata about but you might really want to view an HTML file as HTML so you can switch over and do that and continue to navigate around this is the nav protocol in action not all that stuff is not immediately realized right you can get to that stuff and then as you go basis so let's look at classes closure has for a long time had a reflection a data fication of reflection but it was not a generalized state of data fication of reflection right there's closure lang reflect or something or closure reflect Java at the end remember the namespace that data files reflection but it wasn't a general facility in 110 we added three or four lines of code that link that up to the general facility of data Phi and now when you get to a class you can drill into a class you can look at the base classes you can look at the members of the class you can drill in and this is all data and of course at any point I can stop and click on deaf as and deafness as a var back into my repple and continue going so this is designed for interactive exploration at the repple as you go I should mention by the way that I've been sitting here and typing things in in this UI control but this is integrated with however you launch your repple and however you want to run your development environments so this is connected to my Emacs environment so if I type something over here and send it if I had it in closure mode and sent that then we would see that show up here as well so this is you could conceivably use this as a place to do closure but I use it this actually forces me to two monitors sometimes and so I have to actually sit at my desk instead of at my couch so I'll have one monitor with rebel up and running and then another monitor that has Emacs or cursive or you know whatever your choice of development environment is so let's browse some namespaces that closure 110 also datafiles namespaces so when you look at a namespace you can see the public's and the imports you can drill in and look at the actual functions in the namespace one of the things that's it to do is to move the doc string up to the top and its own little widget the doc string is kind of a special place in metadata but that's a relatively small ad so we can navigate around to all the VARs in the system but what happens when I'm in the namespace world if I nav into the imports which are Java classes well now that I'm in Java classes those things are data fide so now I'm navigating Java classes so there's a compositionality here right by making it a general facility that recursion can go back and forth it can bounce back and forth from recursing on closures metadata to recursing on Java's metadata and if you add data fie and nav on something that you have that gets to play as well right that will work automatically and if your thing refers to a class then you can keep going if your thing refers to a closure namespace then you can keep going so that's all pretty cool but of course eventually you have stuff that you put in a database so it took about 15 lines of code to add nav support to the de Tomic client library and so there's a new release of the new helmet client library that dropped yesterday or the day before that implements these new protocols in nav metadata which means that when you're running with that enabled you now have rebel can recognize an ID in DES Tomic and say oh if you nav to that I'll pull it so I'm going to let's run a query here and of course this query is connecting to it's connecting to nowhere because we don't have the socks proxy running because it fell over so this is actually connecting to date on Macleod to run this query there we go that was actually pretty cool right the query was in a back off and retry loop on the client because he actually couldn't connect and then it got through and so now this data the query that I ran here is pulling out all the releases from the musicbrainz database so that's a single query but now when I drill into the query results if I actually get to anything that's a DB ID and they have into it I can just keep going right because the query result has metadata attached to it that says in order to have two things go back to the database and issue another query you go back to the database and issue another pull so again this is hand over hand and it composes with everything else I expect that I'll be sitting down with shell core field later today deciding where we should put data Phi for closure Java JDBC so you can have this same kind of capability when you're using a JDBC datasource but if you build something that is a data oriented thing that isn't fully data or is large or infinite or is navigable then you can play in this game too and you can build tools that will consume this all right let's talk a little bit more about how those pieces fit together and how everything works first off what's the name mean rebel is read eval browse loop so that virtuous circle of data Phi plus nav plus meta database nav of things coming out of eval we're calling browse and it's a generalization of browsing that you do on the web and it's a generalization of laziness that you do when you're walking seeks enclosure so we've looked at rebel so let's drill in and look at how hard it is to play this game so let's look at the hundreds of lines of code of the data Phi and Navarrete occults so here's the data fide protocol I pulled the doc string out for just to make it more readable on the slide so the data file protocol says how do I date a data find myself and everything has a default representation of identity so everything can everything can play no worse than before and rebel and a whole cottage industry of other tools starting shortly after this talk perhaps can use data Phi to say give me back a data representation the original undefined is placed on that day defied representation as a metadata key so you haven't lost anything all right one possible thing you could say is well what I might need to get back to that thing so it's still there you don't lose that when you go through data Phi and so here's a couple of examples this is some of the data set point data fysop ort that's been added to closure in the last couple of weeks there's data Phi of refs and the data Phi of refs just pulls through and gets the value of the ref and data Phi of closure namespace that code right there is what it took to add generalized laziness for namespaces and closure all right claw the namespace has an API it's not data it has an API that API includes NS publics in s imports and in s interns this code is sitting there in the data file names facing closure so you can look at this to sort of get an idea of how to do your own data fication so now let's look at nav nav is trickier nav is trickier because you have to think about you know what's my chunk what am I going to bring back what am I gonna bring back next how many how many move forward and data Phi is likely to be done on actual things that are not data but nav is likely to be added via metadata to something that's already data but you don't want all of it right so you have a chunk of it and the nav will tell you how to get to an additional chunk so what nav needs to let us do is say we're not going to consume all this at once but we need to choose where we go next and we need to keep track of what we're doing right we don't want to lose that back and forth in the browser where I can slide things to the left slide things to the right and there are tons of things that do this this talk has already showed all four of these things as examples but I'm sure you'll think of others file systems closure namespaces databases of any kind and the web itself all these things are generalized lazy things right you don't want to look at them in their entirety but you want to be able to move around them and continue to have scissors continue to be able to work with data as you go so what does it take to make something navigable you implement implement the navigable protocol on some container ish thing call and the nav protocol returns for a given index and value a transformation of that value and so this is where I've been tweeting about the revenge of the metadata right this is what it looks like for for working on an arbitrary database style data source right a data source produces a query result here the query result is some ID and some details and some other ID and additional details but that query result also has metadata which has a namespace nav key which has the function that implements navin says how do I go get the next thing when you go get the next thing you have to put metadata on that as well so this is a hand over hand process right the top-level collection has metadata that says how do I get to the next level collections as part of navigating then you can say at the next step how do I get to follow on collections after all that and again all of this composes with other people's now have efforts as with protocols generally you should extend protocols to things you control right so it's a thing you made you should extend the data fide protocol to it if it's not the thing you made you probably should talk to the person that made it about extending the data five protocol to it and this is the entire implementation actually half of the implementation that's in the atomic and it shows you the trick right if the thing I have coming back from a query is a collection then I'm going to add metadata to it that closes over that atomic database right so that pound nav eyes DB % 3 that says navigate into the V right that's the third argument to nav and then the thing that's not shown here is maybe pull next which is a little helper function that calls de Tomic pull but you would do something similar to this to any other arbitrary data source that you want it to work with so that's rebel rebel is written with Java effects it is an in process thing so it's integrated via Preval into your existing rebel with your editor I can't wait to see what kind of tool closure script people will make with this doing something that is integrated with that workflow rebel allows you to choose from generic data browsers and viewers one of the things that's a little bit difficult to show partially because of just the size of things on the screen is that that selection of vouchers is based on predicates so these viewers and browsers and this is true of the browser side as well so as we navigate into things we may have I just issued a bunch of the atomic queries navigating around let me back out to something as you navigate around both this browser although usually there are fewer choices for browsers because browsers have to know how to navigate into things and viewers these various choices are made possible via predicates so rebel has predicates that say we this thing makes sense viewed like this one of the areas that is ongoing work is memory system for your choices so right now there is a memory system that remembers your choices I would like to make that memory system even smarter and have it know about your choices in contexts better so that once you say you know I want to view this like this you know that's remembered in such a way that there's not a configuration step for users right in earlier drafts of this it was like well you could configure you know you don't want to do that right you want to have this thing be smart and say you know when I went in when I was in this context before this was the viewer I wanted and have this context before be a flexible notion that anticipates that what you want so that's an area of ongoing work rebel is master-detail it's got this toupane notion where the left pane is some kind of collection ething and the right pane is a place you could drill into and go and has more details about that another way to think about it is that it is a browser that always has next and previous enabled my previous has worked my way back up to the top of the tree and next is you know I can choose to drill into and see in advance of drilling into various pieces of data rebel is extensible via data Phi and nav and and it's really important to take a moment and reflect on doing this in a general way data Phi and nav need to be part of closure and the protocol metadata need to be part of closure because that's what allows this composition and generality across the entire ecosystem rebel doesn't need to be in closer it's its own thing it can do what it wants but all these pieces play together because we have common protocols where we can meet and this is a stunning example of reuse I just want to double underline that because a lot of times you know when people are coming to closure people will have arguments about you know look closures more concise than Java for doing some job right instead of writing 30 lines of Java code I got to write five lines of closure code but this kind of thing this kind of abstraction and reuse is more like an order of magnitude of code that never gets written it's not about some snippet of code somewhere that you made smaller it's about whole fields of code that just go away and are not needed because of the identification of a common protocol this is design work thanks rich [Applause] and I'll just tease a little bit this slide shows a bunch of different things that you might navigate including data and closures metadata about namespaces and your host platforms metadata classes and file systems and the web and databases david shlonsky will be talking about doing all of this and more for AWS later in the conference so and all of that will compose without anybody writing any code all of the work that david has done with AWS will compose directly into the system via data Phi and nav so get the bits if you want to play with data Phi and nav and the metadata protocols go and download the most recent are C of closure closure 110 are C 2 and if you want rebel stay tuned when I walk away from this podium I'm going to go up to my room and publish the bits and you can download this and play with it rebel is free as in beer and I hope that people will enjoy it and be inspired by it and use it in their own projects and build other things also with data Phi and nav thank you [Applause]

Channel: ClojureTV

Views: 20,859

Rating: 4.9441342 out of 5

Keywords: clojure, Datomic

Id: c52QhiXsmyI

Length: 33min 5sec (1985 seconds)

Published: Thu Nov 29 2018

Reddit Comments

[u/halgari]

Just for fun I looked at the license, expecting it to be fairly standard "free but closed" language, but saw one line that bothered me, so I created a issue:

https://github.com/cognitect-labs/REBL-distro/issues/1

Until that's resolved I'm not sure I *can* use REBL.

[u/alexdmiller]

https://twitter.com/cognitect/status/1068223505764950018

[u/NewazaBill]

Amazing stuff! Now the real question: will there be a dark mode?

[u/fullhalter]

To start messing around with the datafy, nav, and metadata protocols you'll have to be on Clojure 1.10.0-RC2, so

 [org.clojure/clojure "1.10.0-RC2"]

If you're using Leiningen. From there just:

(:require [clojure.datafy :refer [datafy nav]])

Stu said that an early access of REBL should be available later today, but didn't say where. I haven't been able to locate it yet, but I'll try to update this comment when I do.

Edit: Here's the REBL Github and the REBL download page on Cognitects site. Thanks Alex!

[u/Ewen-g]

Cool stuff !

I wrote something a bit similar for emacs.

[u/zqvt]

is this essentially Pharo for Clojure?