Amjith Ramanujam Awesome Command Line Tools PyCon 2017

and hello and welcome to another talk here at PyCon my name is I'm Jeff Ramanujan I'll be speaking about command-line tools if you have any questions or feedback about the stock you're welcome to reach out to me on twitter my twitter handle is at i'm just are for my day job I work at Netflix I'm part of the traffic engineering team where we get to move enormous amounts of traffic between AWS regions at peak traffic we sometimes handle over one-third of entire Internet's traffic it's just a lot of fun but I'm not here to talk about my day job I'm actually here to talk about a couple of my side projects one called PG CLI and the other one called my CLI they're both command line clients for data bases Postgres and and my sequel respectively and this talk is actually about some of the design decisions that we as the core team of BG CLI and my CL I took and in order to overcome some of the inherent limitations of command line interfaces we took a lot of inspiration from existing good command line applications that were well designed and I'm going to go over some of the some of the other tools as well that will be covered and where we took pens for Asian from the story actually begins about 20 years ago the protagonist was typing away furiously on the on a computer terminal on the on a cutting edge operating system called ms-dos at the time he was typing one character at a time because he did not know how to touch type at the time and he finished the command presses the enter key and looks up at the monitor eagerly waiting for the results only to realize that he made a typo and it resulted in an error I'm defeated by the setback he proceeds to begin typing that command one more time hoping that there is not going to be a typo this happens for a couple a couple of weeks until his teacher introduces him to this concept of history navigation the way you do this is you press the uparrow and suddenly the previous command that you typed comes back up and you can edit it accordingly and suddenly you're able to fix the fix the mistakes without having to type the entire thing one more time it was a he was euphoric the kid was just G billion he had lots of hot chocolate that day to celebrate the protagonist may have been me later on when I was in grad school I was introduced to the Linux and I was pleasantly surprised that bash kept the convention of using the apparel and down-arrow in order to navigate history and that was that was great and by now I have actually learned how to touch type and I was typing at a furious blazing speed of 50 words per minute that's right a pair programming with a friend of mine and I noticed that whenever he typed any file names or path names that happened to appear at a tremendous speed it was it was beyond humanly possible and so I was surprised I'm like how did you do that and then he introduces me to this concept of tab-completion because now you can hit a tab and the shell automatically knows what you have you know whatever prefix you have typed it'll try to match a file name to that and it'll automatically complete complete the rest of the name for you oh that was another joyous day much chocolate milk was had and the moral of the story here is is not that not that command-line tools have awesome features which they do the the fact is those awesome features are not easy to find it's actually pretty hard to stumble upon these features unless someone introduces them to you by by being a new user to a command-line interface it's not easy to find these features organically you are either reading a man page which is a different program from the program that you're trying to use or you have a good mentor or a friend who who tells you how to do these things let's contrast that to a GUI application whenever a new feature or a new version of your application is released if there is a new feature in that then there is an icon or kind of a toolbar or a menu item that is listed and being humans that we are whenever we see something new where we tend to kind of poke at it until we find out what it is so how can we how can we bring this idea of discoverability to command line applications the the two that the two features that I just showed you which is history navigation and tab completion I'm going to take that as examples and show you what we could do to make those two features more discoverable in command line apps and that brings us to the first application I'm going to show you how we solve this specific problem in pgc Li have you ever wondered why we call that tab completion why do we need the tab key in order to complete that for example I'm going to launch PG CLI I'm launching a particular database here and I'm going to type a select statement and try and see you know how we could complete that as soon as I start typing I am presented with all the completion options here I have not pressed the tab key and this is how I think we should be making this more discoverable where I could type select star and again it provides us with all possible completions and we can start typing those things we did not use any tab key in that in that scenario I'm going to switch over to a different program that we are going to look at which is called fish shell which is a modern shell I mean the word modern is it might not be appropriate because it's been 20 years since it came out but it's an alternative to bash and Z shell well this is the feature that I'm that goes even beyond the history navigation of up up and down arrows if you've used bash there is a special keyboard combination called ctrl R which allows you to search through the history so you press ctrl R you type up some of the partial string from the command that you type in the past and it'll try and bring up that that command from the history by searching through the history and this allows you to navigate history in a much more random manner rather than the linear manner in which you know you you are used to using the up down arrow again it took me a couple of years before I found control-r and once I found that there was I would always prefix our I would always start my commands by typing ctrl R and try to SSH into the into the machine that is attached like last week or yesterday or something now official kind of obviates the need to start every command with a control are the way the solves this is as soon as I start typing it will start to search through the history and it will start recommending things for me and it is automatically suggesting things for me and I could just press the right arrow and make that happen or press ENTER and and that command will be executed so I could do work on and I have been working on prompt or make a new virtual environment all those things as soon as I start typing it pulls up that command that was similar to the one that you are trying to type from history and it starts suggesting that this is a fantastic way in which discoverability was solved in fish shell for that particular feature that is we looked at some specific examples but in general the idea behind discoverability is that be a little bit more forthcoming do not hide your features behind a special key because those special combinations are going to make it undiscoverable so if you want your features or your make your program more discoverable just be a little bit more forthcoming with your features the next item that I would like to talk about is user focus and what I mean by user focus is that whenever you have whenever you are implementing a application or a program or a feature the users always come first think about how you could make your program the most intuitive for the user absolute absolutely powerful for the user don't think about how much harder is going to be to make the implementation of it the implementation should always come next in order to demonstrate this I'm going to start with my CLI one of the other programs that I showed but before showing my CLI I want to show you my sequel and my CLI is a is an alternative for my sequel so my sequel I'm launching a database launching the repple here with the database I start to type SEL and I hit a tab and nothing happens here I expected at least some kind of tab completion let's redo that again by doing all care all caps on everything uppercase and I hit a tab key and now it actually autocomplete for me there is a simple little tweak all we have to do is make that auto completion case-insensitive and it would be that much more easier for a user to use it and it is a simple thing that the the programmer could have done but it but they apparently didn't and after the select start from and I hit a tab key and it asks me if I would like to see all 811 possibilities that I that it's going to offer I certainly don't have 800 tables in my database I say yes and I'm presented with every single keyword that is available in my sequel let's contrast this with how my CLI handles a similar scenario once again as soon as I start typing it's going to start suggesting things and you'll notice that it is not case-sensitive so I could do select star from and I hit a an a and it automatically suggest only tables that come after the from keyword because it knows that these are the tables in that current database and it goes even further when you choose where it'll only give the columns inside of that table because it knows to scope inside of inside of the show only the columns inside of that table thank you [Applause] when I first got got started with my CLI I had actually taken the approach of doing what my sequel was doing which is show all the key words and let the user type them out and try to figure out you know from the from the menu but then later I figured that actually being context sensitive about the completions will make it that much more powerful for the user to use but at the time and still now there is no open source SQL completion engine it was ridiculously hard to implement it took me another two three weeks before I could actually call that then complete and it was in a broken state I mean it was it was not a very well polished one and thankfully there are now smarter core and developers for my CLI who have done a much better job of doing this completion so the point about the user focus is that make the users always come first whenever you are implementing a new feature or the program think about what the user wants think about how to make it most intuitive and absolutely powerful for the user do not worry about implementation implementation comes next you can make that happen I kind of saved the best for the last be Python is the favorite interactive shell that I absolutely love I've taken a lot of inspiration from this and stolen a bunch of ideas from here you'll see why in just a second once again I'm going to start out with Python to show you what could be improved I'm going to import a module a type I NPO and I hit tab and a tab key gets inserted I don't think there is any business and a tab key in a Python code at all ever thank you so I've imported requests and I wanted to request that and I hit a tab key again I'm trying to see what is available in requests now how can this be improved well let's look at what be python does here i do IM p and it not only gives me the the list of items that could be completed it also does fish fish style autosuggestion where i have typed this command before so i could just press the right arrow key and proceed and i do requests docked and it gives me all the available things that i could do it I could do a post command if I wanted to sort of get and if I open a bracket here it shows me the different arguments for that particular method and not only that it actually shows me the doc string for that method so I never ever have to leave the terminal in order to find out how to use this particular method I think this is a powerful powerful powerful tool and one of the common things that I get a push back on from users when I show B Python is that of course the pythons default thing could do most of what you have shown all I have to do is copy this particular snippet that I got from stack overflow put it inside my Python RC file and suddenly if I hit the tab key it will do some of the things that you actually showed and I don't have to install a new program my counterpoint to that is that configurability is the root of all evil I'm actually paraphrasing things from fish shells design documentation here which have taken a lot of inspiration from one of the things that is mentioned there is that whenever you'll have added a configuration option to your program what it means is that your program was too stupid to figure out what was best for the user by definition or by extension it's also calling the person who implemented the program stupid so if you I do completely understand the configuration configurations are essential for some situations because certain things are taste based they are subjective say for example color scheme picking a color scheme I mean you might think that solarized is the best default and I agree I mean it is a good sane default but some other person might want a lighter color scheme or a much darker color scheme in that case I think subjective options are okay so if you are going to add a configuration option make it a one where the program really can't figure out what the producers taste our tastes are alright we looked into three specific issues with command line applications discoverability user focus and configurability and we showed some concrete examples from some of the existing tools such as fish shell B Python PG CLI and my CLI at this juncture you might be thinking well I learned about these tools I'll start using them in my workflow but it looks like these are some advanced features these are going to be hard to implement when I'm writing my own interactive shell either for my internal use or I plan on releasing something to use the users these things are much harder to implement and these things are you know you might have competing priorities like deadlines or you might be thinking this is only for internal users so they're going to be forced to use my tool anyway so I don't really have to care I've done that before so it's not it's not that uncommon and I completely and I completely want to empathize with that sentiment but what if I told you that implementing all these features is actually not that hard what if I told you that they can all be done in under 10 minutes with merely 10 maybe 11 statements in Python in fact this is a checklist that I that I typically use whenever I am implementing a interactive shell which is you must always have a persistent history which is just because you quit a program and you come back to it it doesn't mean you have to start from a clean slate because obviously you have type you know some important commands in the past and you like to be able to recover those so having a persistent history is a great thing the ability to search the history either using a command our special key which is common these days and people power users of shell already know about it or using the fish style suggestion where it goes above and beyond and doing that an eMac ski bindings this is another hidden feature maybe not so hidden anymore because a lot of people are used to using bash pressing ctrl a will take you to the beginning of a line pressing ctrl e will take you to the end of a line and ctrl P and ctrl n or equivalence of up arrow and down arrow and these are these are default key bindings that comes with any any shell bash z show and even fish shell everything and I think it's essential that anytime a user is interacting with an interactive shell they have come to come to expect these things to be available so that's good to have paged output is when your output that you are trying to print is much larger than it's going to fit on the page then it is good to send it through a pager so the user doesn't have to scroll up and down it's just a convenience and the last three are the items that I kind of showed just a minute ago which is auto completion being able to automatically trigger them without having to type the tab key it's a it's a very nice to have having minimal configuration in fact if you don't have any configuration at all much better and adding syntax coloring so I did not show or I don't know if you notice this or not but all the shells that I showed today my CLI PG CLI facial and B Python all of them have syntax highlighting automatically so as soon as you start typing them it will appropriately color those commands for you we're going to implement most of those things in the checklist using just a single tool or a single library called prompt toolkit I'm going to show you how and we're going to actually implement one called a repo a rebel stands for read eval print and loop which means you're going to read a user input evaluate what the user what needs to be done with the user input print the result of that of that evaluation look back and start taking user input from the user again I believe I still have 10 more minutes left how many of you would like to see me do this live and how many of you want to just alright do it live okay so we're going to create something from blank slate there is nothing in this file I'm going to start from scratch first thing we want to do want to be able to do is read an input from the user and typically if it is just a pure Python program without any library is imported then you know standard library comes with the input command or the input function or raw input function but we're good we are not just building any any repple we're building an awesome rebel so we're going to take the prompt function that's provided by prompt toolkit so I'm going to I'm going to pull in the prompt function and it has the same signature as the input or the raw input where you give it some kind of a prompt and the whatever the user has typed will go into the InP variable and to keep things simple our eval is going to be a no op and what I mean by that is whatever the user has typed we're just going to print it back to the user so it's just an echo rebel if you will and we obviously want to loop back and make this into an infinite loop so we'll just put a while loop there okay how does that look like so now we have a rebel that can take input from the user and it can do it can do just echo it back to the user it does not have history it does not have any kind of auto completion no syntax highlighting nothing but we can fix that obviously if I press control-d it's exiting with a trace back but again we can catch that exception and do something useful with it but we're not going to do that right now for the sake of time so the next thing we want to do is add some history to this repli'd up we just built and again prompt comes with everything you need in order to build an awesome raffle and I'm going to do I'm going to add a history up to it and import file based history because we want this history to be persistent we don't want it to go away when the when we've quit a particular session and store it in a file let's call it history text ok so now I type a B C D F I can press the up arrow down arrow and I can even do select from and if I do a ctrl R it allows me to search things and that's all it took it literally seven lines with white space including and that is having persistent history and it also comes with the Emacs key binding so I could do ctrl ctrl a ctrl e control B ctrl F do all of that stuff this is all built into the prompt built-in to prompt ok this is already a good working repple that we could use but we're going to continue to make this awesome so the next thing that we talked about was ctrl R is nice and all but we just talked about making it more discoverable so we can take the idea from fish and let it automatically suggest things from history that matches this scenario matches the command that we're trying to type and again it comes with autosuggest and you can also just from various things but we're going to autosuggest from history okay and now it's actually Otto suggesting right there and I could select that and this is coming from history and again just one line to add the fish style auto-completion the next thing we talked about was we don't want to be using tab keys for completion we want it to be automatically completing the things as we start typing things once again prompts toolkit to the rescue I can provide it with a completer and since I'm typing select statements let's make this into let's say that we are writing as I'm writing an SQL Lite CLI so we want to be able to provide automatically complete SQL keywords in this case so prom toolkit comes with a lot of different completers and I'm going to pick one called sorry I'm blanking the name word completer thank you okay and I'm going the way word completed works is as soon as you start typing it is going to do a substring match on a list of strings that you have provided as its corpus and it is going to start providing the suggestions from that so we need to instantiate a word completed object and initialize it with a bunch of over the list of keywords so let's call our new object as SQL completed because that's what we're trying to build and it's a word completer and it takes a list of items and let's say select show let's keep it simple just just four items maybe like from and we're and we made a big deal about how my sequel was being all case sensitive with its completion so let's ignore case thank you thank you and we pass the completer to the prompt function let's see how that works so now I start with the statement and right away it is trying to suggest things that I could be doing and it still has fish style completion that I could try and try and use here and again that is all it took to actually build a ripple that can autocomplete I can do auto suggestion like fish and that can do history navigation with the up/down arrow and all of that another item we kind of briefly mentioned is using syntax highlighting in Python there is a powerful library called pigments if you don't know what pigments does it does syntax highlighting for your code and it comes built in with syntax highlighting for a lot of different languages since we're dealing with SQL here we could pull the SQL syntax highlighting lecture and automatically feed it to prompt and prompt book it has built in a way where it will automatically work with the pigments lectures and you could do autocomplete a syntax highlighting for that so let me feed the SQL lecture to this obviously I need to import that so that is coming from pigments so I am in fact using more than one library here but I think will will not worry about that okay now I have a select statement with Auto completion and that has syntax highlighting here thank you by the way all the code that I typed is available here in the slides and I will be posting the slides on my Twitter account pretty soon and this is our checklist that we started with and we finish everything except for paging through the output if you are interested in knowing how to do the paged output check out click-click as a another command line a library for python that allows you to echo any output through a pager and it is cross-platform compatible so it'll work right off the bat in Windows as well as in Linux by the way prom toolkit is also completely cross-platform compatible so you could use this in the program that I wrote will also work in Windows as well as Linux these are some of the resources that I shared today PG CLI my CLI says she'll be Python interpreter and prompts Tolkien if you have any questions or comments about any of the things that I've said in this talk you can reach out to me on Twitter at i'm jeff are the people that i've listed on the right Jonathan slender sees the author of prompts toolkit he's here at at PyCon and first row actually and I had Thomas Bollinger he is a core developer in V Python he's also here at at PyCon and Edina she is the lead maintainer for PG CLI actually I've given the range to Edina and she's doing a fantastic job she's also here at Python and she's somewhere here in the talk so feel free to reach out to any of them and they'll be happy to help that's it thank you very much goodbye [Applause]

Channel: PyCon 2017

Views: 26,664

Rating: 4.9451218 out of 5

Keywords:

Id: hJhZhLg3obk

Length: 28min 42sec (1722 seconds)

Published: Sat May 20 2017

Reddit Comments

[u/j605]

That was an amazing talk!

[u/amjithr]

I'm the one who delivered the talk. Pleasantly surprised to see it posted here. Happy to answer questions.