Use curses, don't swear

good afternoon I will be talking to you about using curses in Python and also a third-party library or wid this will just be a high level level overview of terminal manipulation and Python those with n curses experience it will be review for those without it you might learn something hopefully so a little bit about me I'm a web developer and the IU logo is a joke because II's is a difficult browser I am a PHP turned Python developer as well I've done PHP Java JavaScript and you know some various things like that I enjoy learning all things low-level so if it's a language you have to manage your own memory I'm probably going to like it and if you deal with pointers I also probably like it for some reason the pictures are just representative of low-level things really nothing more just that diagram on the top and assembly on the bottom and I also love computer history it's a Commodore pet on the top which is came out in 77 and an Apple one which came out a late 70s I believe if in any period of history that I could go back to it would be the late 70s that really 80s because it was just a lot going on not that there isn't now but there's a lot that was very new that we take for granted and I just like to stay at that level as much as I can so where do I work at work at cars.com specifically in new cars' calm in sunny Santa Monica it's a top-notch company great management great tech department we're also a PyCon sponsor we're very glad to be part of the community very vibrant community of humans and we are a Python post great shop as well migrating from PHP and MySQL and an assortment of technologies mixed in there but eventually our goal was to be is Python as much as possible because it's just a great language we're awesome place to work and we are hiring so if you're interested in working in Santa Monica gentleman down here named Michael is our hiring manager you could always talk with him or any of us here in the front row we're all from cars except for the gentleman that introduced me though he is still nice and valid okay so before we get into the rest of the actual programming we need to talk about video terminals a little bit just set the stage your terminal identifies itself through a term variable so this one thinks it's an X term that's capable of 256 colors which of course means that it thinks it's capable of handling colors the teapot command lets you query the term info database that most computers have installed and pretty much all of them now to get an idea of terminal capabilities this one is also capable of moving the cursor around or called the direct cursor manipulation and let's see what happens when we try in vim ignore the top line we try and then to use colors apparently it works and it's also government-approved so let's switch that back and get out of there and next we will try and issue some raw escape sequences and change colors on the command line on our own without the aid of anything but just the the keyboard isn't it amazing how I can talk and also type effortlessly so here I'm printing out a warning message what's happening here this is an escape sequence and these are literal characters you can get an escape sequence by just pressing ctrl V and then escape and then the rest follows it's a command so this is changing the color to blue and this is changing the is changing the video mode to underlined and then we're printing out the word don't and then we change it back to default white on black we print the rest and then we change it color to red or actually this video mode reverse and then color to red or the other way around can't remember we print out red and then we reset it and press the wrap and and print the rest so if I wanted to how could I find out a terminals full capabilities if you want the bleeding edge viewer blue blistering eye view if you will this is the actual entry for X term in the term info database and there's quite a lot going on there and I'll be honest I don't know what most of these things are but it's very capable as you can see this is an escape sequence such as how you represent it switch my notes over so there's other types of terminals and in fact which is why why we have curses in the first place is because we have so many terminals so if we tell this terminal that it's a different type of terminal we can see what happens so we're going to tell it that it's a vt100 is anybody familiar with the vt100 nice so clearly it now thinks it's a vt100 and vt100 is not capable of colors and it is capable of direct cursor manipulation though so if we use them and we change the color scheme nothing changes because the vt100 cannot display colors or so it thinks great and then we can see what this terminal is capable of and you can see it's a pretty respectable list but it is definitely less than the X term which makes sense because the X term is an emulated terminal so you're probably going to be able to do everything and then some so we don't have any hardware restrictions in that case but this one this one came out ooh 70s early 70s so you know it does it does what it can there was also another type of terminal called the dumb terminal and that was very very limited if you're familiar with them teletypes it's very similar to those so we can also tell the terminal that it's a dumb terminal so when we check if it's capable of colors no it is not it's also not capable of cursor manipulation what is this terminal capable of not very much at all that's like four things so you can pretty much type a command and press ENTER and that's about all I can do you press ENTER a lot if you want to try and clear the screen but it's not going to do much for you so I'm going to use midnight commander this time because vim is a little too nice and it will try and render itself even if it can't so if you try and run midnight commander it's not even going to start because you can't so this is just to fill out in details on why we what kind of terminals we have and and their different capabilities if you want a more complete list of terminals this is everything that's in term info right now so currently if you write an in curses program you're actually writing a program for all of these terminals well it will run on as many as it can as you saw from midnight commander okay so let's skip back here and talk a little bit more about video terminals first terminal came out around 1969 it was a data point 3300 at least that's one first one I could find it offered basic line editing like backspace finally clear line you can clear the screen you could move the cursor to any point on the screen it was quite literally just a monitor and a keyboard attached to a computer via an rs-232 cable or connector use the ASCII character set which was improvement over the body code which is a 5-bit code previously and it ran at about 300 characters a second or 300 baud this is a picture of the VT 100 which we saw its mode in X term it was one of the most popular video terminals back in the day product of digital corporation it ran at 24 lines by 80 columns which is a default setting and pretty much every terminal out of the box today so you can see it's legacy also there's a lot of support for its escape sequences today still most emulators still have a VT 100 mode so talking to a terminal the kernel would handle all the low-level transfer data details which you probably familiar with would receive input from the keyboard would send it through a line discipline which we'll talk about a little bit and never send it to the user processes and that's actually still how it works it just we don't have any hardware terminals anymore or where they're not very widely used I should say the monists switches so there were no standards pretty much every vendor that came out with the terminal had its own set of escape sequences this is from the VT 100 ACMA 48 came out in 1976 tried to standardize that 108 pages of very readable codes and explanations and on top of that term cap came out in 1978 I don't have a picture of that but it basically was a precursor to term info and it was allowed a program to say what can I do with you terminal and here's a line discipline picture of it it's I like to say it implements the bridge design pattern because it bridges the gap between the hardware and the software you can have n number of different hardware devices and number of software processes they can all talk through a line discipline it is glue code it is not called outside of this context and it can't call anything else really so it's not alive it offers two modes raw and cooked if you open up a terminal you are in cooked mode which means you type a bunch of stuff press ENTER and and then it executes if you press control C that becomes a signal the kernel emits a signal you know SIGINT in that case if you're in raw mode everything that you type including special sequences goes straight to the program is not interpreted at all so introducing end curses finally we come to it originally developed for BSD back in the day it actually borrowed code from VI which is my preferred editor of choice back then VI had a lot of terminal like cursor manipulation features and so encourages the courses at that time took from that and now then borrows from and curses so it's a good relationship originally used term CAF now uses term info it was moved to system 3 UNIX system 3 was not read was not ported because there were issues was reimplemented the game's new features like color and being able to draw more draw lines easier and new video modes like bold and underlined and then curses became p curses due to licensing issues and later became n curses and so we come to today so why n curses well by now you have the picture that there were a lot of terminals and there are a lot of different ways to talk to the terminals so the way term info standardized our querying of a terminal this standardize the way that we can write programs for it you write an anchor susp program and it's going to run everywhere that term info is supported basically or previously term cap there are convenient abstractions like their windows that you can put you don't just have to move the cursor draw a line you know move the cursor put some text etc you create a window pop it on the screen refresh and and you're you're going it also had a standardized API which is goes right along with its portability so the philosophy the tools are provided but you are on your own they give you one primary object which is the standard screen which represents the entire screen size you have full control over that you can have put form form elements windows colors and they introduced a new mode which is character break which sits between raw and cuts so raw I just talked about those you remember those character break it's going to break and every character but it's also going to interpret character your sequences like control C control Z and all that stuff so and also you were right you had to refresh the screen manually in the right order which you'll see in the program so let's look at a program this is the program we're going to build here it just gets random quotes at this point just Chuck Norris quotes if I press R gets a new quote hopefully somewhat quickly it is going out over the internet could have been something I fixed beforehand so and the more you do that the more it gets you quotes sorry about the swearing I can't control Chuck Norris unfortunately pretty simple but let's build it out and see what it looks like I hope you all had a chance to read that because I definitely skipped ok so we are going to start with a blank screen we can call this screen and ocean right now there's nothing living in the ocean but we will fill that ocean with code that's an inside joke for people that I work with based on a conference a couple years ago so here's what we're going to do I'm going to fire up vim and we're going to import the cursus module we're going to start building it out this is the code that's going to get us our new joke you can see it's just grabbing a string off line and then translating nothing big please don't judge the code it is definitely simplistic then we're going to create our screen this is going to issue some escape sequences to the screen the terminal it's going to create some internal data structures all very boring and then we are going to tell the terminal don't echo my characters as I typed them we're going to set it in the character break mode and we're going to hide the cursor and then we're going to check this have colors okay if so then start the colors and then if optionally if you want to receive keys like f1 f2 all those those are multibyte character sequences so you get to tell curses that you're expecting those and to let them in unharmed oops back one ok and then we have to tell curses we need to use these colors so this is foreground and this is background so for foreground background and we have three of those so we haven't actually built the UI yet so now we're going to add the top line so the first line here adds the random quotes in Reverse and then it fills the rest of the line so the effect of these two lines is basically pops the random quotes in and then it finishes off the line and then we are going to put the menu at the bottom which basically does that this this demo here doesn't have color there but these two lines are going to add color and it's very low level you can see you're going down to the last line 7th character you're turning on bold and color pair 2 which is green and you're saying just render it re render that in green same with the cue so that's the are same of the cue it's going to be red and then we're going to create a new window and that's the quote window right here and hopefully you can see that border we haven't actually put the border around it I'm just doing that for visibility next we're going to do the little text window which is invisible watch which is why it's dotted like that but that's where the text is going to go and that that way we can rerender this inner part and not touch the outer part because if you if you refresh things in the wrong order you can actually overwrite your own Changez then we're going to add some text right in there I'm sorry I want to work quickly and then it's okay and then we're going to create the box which actually does this border a little quicker and then we're going to refresh the two data structures notice we do the standard screen first and then the quote window if we do it in the other way the standard screen will cover the quote window so it's a little tricky and then we're going to update the screen all at once then we're going to do our event loop this is just the basic event loop we're going to catch the input every time if it's an R I'm going to clear the text window we're going to say getting quote and refresh it then clear it and then put the new joke in if it's Q then we're going to break out and at the end of the loop of each loop we're going to refresh from the bottom up so screen quote window text window and then draw the screen again and the reason you do this is so you avoid flickering and then if we've broken out of it we're going to reset the terminal get it out of character break mode turn echoing back on and then show the cursor and then we're going to end and that's all you have to do in order to write a simple cursus program so now in the little bit of time that I do have left let me save this in front of you and then talk about a widower wit is a third-party library that does a lot of work in curses does a little differently we're witness pronounced or wid it's German for ancestral so it's basically ancestral widgets which is kind of a play on how old n curses is it's got a layered architecture which is very much different from in curses so you can see you've got display modules widget layout which is where your UI is and then you can have different event loops that actually control it because it's layered you can do things like this by default you have a raw display in wit which handles a console but you can use the curses but because it uses raw it can actually do more things like the high colormode 256 colors which you can't do with curses that I'm aware of you can also have a web display so it's kind of like a web server but that's not quite developed yet and then an HTML fragment case you want to have screenshots which you can see on their website widget layout is how we create our UI that I don't have enough time to get into how these plays together but there's a lot of detail on how it passes control back and forth but basically you compose widgets it's clearly the decorator pattern in action and then event loops by default they give you their own but you can also use twisted or you can use G events event loop those hook in relatively easily so let's look at a program in Irwin or the same programming error would actually type quickly computer ok we're at 2 X good okay so we're going to import or wood then we're going to setup our pallet and it's much simpler as you can see we just define a list of tuples this is the name and then foreground/background if you want to do something special and fancy like bold just becomes a comma separated value there then we're going to create our header now this is a text widget and we're going to decorate that into an attribute map which basically spreads out the title bar attribute which is the reverse black white on black or whatever it was and that's all you have to do for that and then we're going to create the menu at the bottom so you can use a text widget again and you're going to give it an iterable which has a bunch of different text snippets if you do a tuple like this you can tell it for this one character this many characters use this these properties so it's easier to you know pinpoint yes oh is it oh I'm sorry that's going to be tough because I pre-recorded this has it been tough the whole time and nobody has said anything well I apologize you know what I can do is this does that do anything for you like as a human well I really apologize for that if I do another talk I have to remember that great and then now I can't see the keyboard okay yeah we'll be good okay so now we're going to create the quote box we're going to use another text widget but we're going to we're going to decorate that with a filler widget which sort of distracted me the filler widgets going to give the text widget dimensions because by default you can't just shove a text widget onto the screen because it doesn't have any dimensions yet so we're going to put it in a filler and that's going to align it towards the top and that's going to put a border around it or just like an invisible border on the top and bottom we're going to decorate that into a padding widget which will put the invisible border on the left and right and then finally in a line box which is going to put the border around it the visible border so the same thing that took us upwards of 10 or so lines in curses is actually very easy with the right widgets in or wid or wit and then we put it all together in a layout using a frame which expects a heady a header a body and a footer so we're just going to feed those in and that is all it needs to get started and then we're going to add our joke logic which is the same as before one function to handle all the input because this is a really simple example and then we're doing the same thing but you can see it's three lines as opposed to five or six or whatever it is instead of breaking out we just raise the main loop exit and then we're going to create the main loop giving it the layout we're going to give it the palate and say we get a keystroke send it to this guy and then we're going to run it oh and then we're going to see my mistake and then we're going to run it and there works and let me just back up real quick so stop playing very good okay so that's that's basically that if you want to know more about in curses or or wid then you can look at these links here there's a lot of information rue it is actually been around for a while and I'm sure some of you have used it it's got a lot going for it but I just didn't have time to get into a lot of examples things or so I thought so at this point it looks like I do a little bit of time left are there any questions I know I'm supposed to thought of something it was great he was stirring Lea fast how would you do a timer and end curses a timer like if you wanted to have a clock and it was counting down or something in the top corner you'd have to set up your own loop that would just get down to Python I think I heard somebody saying at in the the booths were still going on like if you're using pyramid and you're trying to do something it's just Python it's like you end up setting up that loop and then you just have to refresh the screen every so often and keep it updated very manual you know you talked a little bit about the twisted event loop integration I could try to but I haven't actually used it yet essentially I notice it gives you lots of features as far as asynchronous tasks and things like that I'm going to sound really dumb if I try and talk about it but essentially it's just a different way to the format responsibilities so sorry I can't be much help in that but they do have some documentation on the website I would point you to this one this gentleman right here obvious question but perhaps with an obvious on so so why would I use cursors to do something these days when I can build a GUI so easily on my desktop right well actually my interest in this comes from my work a lot of the stuff that I do is from on remote terminals and it's hard to have an X server running on those so it's just easier for me to fire up something like midnight and ER and to you know that and as well as hardware that's really limited there's no reason to go around getting an X server running when every you know every Linux install has a console out of the box then if it has a console and then you you've already got a UI and most cases it's sufficient less you got to do graphics then you get again into frame buffers and that's a whole different thing unless you just go the next server yeah basic answer so as a happy Raspberry Pi owner and I'm not using this type of setup and I'd like to since you've researched both types of libraries which one would be easiest to jump into oh as far as curses or or would yes well to be honest they both have their learning curves ruied does a lot more for you you know if a few lines of code go a long way but you got to know how it thinks curses is just much more step-by-step you literally do everything you want to do is step by step so I'd say would will get you going faster but it's good to know how to do curses and that way I've made everybody happy that answer yes uh since terminals are usually ASCII what happens when and curses comes across like a unicode character or something like that good question I know that support for a unicode can be spotted within curses especially at the sea level so I believe Python has some ways to handle that that'd be worth some research I know ruin has enhanced Unicode support like it actually deals what you saw my examples were Unicode characters so curses I'm not too sure there's a library as language basically when compiling courses it actually accommodates that there is an alternative idea of what it is but most installations now come with it so there's support as far as I know just out of the box cool there you go so so as a program writes you know tools that run the command line I tend to use the command sequence the CNC sequences to get the color in show up I've been told you should use in cursors or something else like that but I haven't really seen that I can use these types of setups for you know it's a classic terminal that's just fitting text out but I want color to show up you know the bold or red or green or but that can you do those type of things within courses is or is it just always a frame buffer you show up and everything's lost what you put with the program such is it like you basically still have a terminal but it just has color support yeah yeah so instead of me like knowing all the special sequences I can just say you know it'd be nice to be able to say just you know red green without making my own infrastructure seems like you should be able to reuse there's actually there's actually two ways I can think of to do that that T put command lets you set colors if you're going to just be straight you know going into the script I'm not sure how recommended it is to use that but you can use friendlier names with that also with curses well you saw how you could switch you could switch colors with those color pairs if you're going to use curses there's going to be a little bit of setup I know it does support the scrolling text the logo so it's like being able to say I have you know Python print and I'm saying print something and I can you know I usually spot the color sequences on there basically right where the captain's right so with curses you you'd have to set up your color pairs so it's you'd say I'm going to use these colors overall and then you just say an it pair and you say color pair whatever number it is and then just going to switch to that and then it'll issue those does that get close to where you were yeah I guess what the main thing is at least for my experiment there's ways to be the standard screen and you turn the standard screen on everything on the console disappear to sort of back top right yeah stuff shows up on the screen and once the program ends everything on the screen disappears which seems to sort of defeat the whole purpose of you know terminal redirection and stuff along those lines so and that's where I was sort of thinking about seems like I would have to do it myself because it seems like this is the notion of making out you know a GUI you know console based UI in a sense or so graphical one so you're talking about using it to kind of augment make it easier to to use some of these escape sequences without having to clear everything and kind of take over yeah because I mean I mean it's doing console programming basically getting colors on the screen it's really sort of useful you know it holds prints things out for things that are important especially if you're doing like you know for me like build systems because you know air red shows up and less stuff's going by you can see that red color and see something was wrong right yeah I know it's possible to do but I couldn't give you like a formula right now to do it but I know it's used programs like telnet use it even though they don't use colors I'm going to understand which doesn't clear the screen but it still gives a little bit more control so it is possible to use it incrementally if you will but that's about all I have to offer on that yeah I was just curious um you said that using the end curses library lets you do much more low-level things but using our would gives you the widgets that you can does or would get in your way at all say if I wanted to implement something like a roguelike game where I'd want very specific direct control for each characters and moving them around in particular ways with if I used or would if I if I started with or would what I run into problems because I couldn't get low-level character access right right is it so high level that you can't get down and buy something so two things come to mind one is you like for example or would automatically refreshes the screen for you but if you need to you can drop down and just take the main loop and say draw screen which I did in the example so that's a case where you normally don't have to worry about it but you'd have you needed to do it another case was was the the raw display that would uses by default doesn't support all the older terminals so actually your program out of the box wouldn't run on those you'd have to tell it to use the curses module so you can do that but it's a there behind settings it's not that hard to get to but but yeah you get like I'd say all new terminal functionality out of it and if you have to do something for older ones it'd be a couple more lines of configuration going talk thank you yeah thank you any other questions I'm afraid of actually all the time we have for questions today I would encourage everyone after this to go to the Mission City Ballroom for the closing address and afternoon lightning talks it should be fun as always I'd also like everyone to give a big round of applause and thank you to Sean for his talk excellent thank you

Channel: Next Day Video

Views: 71,001

Rating: 4.9178081 out of 5

Keywords: psf, pycon2013, talk, SeanZicari

Id: eN1eZtjLEnU

Length: 30min 14sec (1814 seconds)

Published: Wed Mar 20 2013

Reddit Comments

[u/koprulu_sector]

This is awesome! Always wondered about various terminal settings. Thanks for sharing

[u/MeanEYE]

As someone who had to make occasional terminal applications I despise ncurses with a passion. Their function naming is opposite from intuitive. Just look at their naming for output functions. To make matters worse, since library is so long winded and annoying to use, they started adding convenience functions which make matters only worse when you are first learning the library because the approach that is taken is to pick seemingly random characters from two or more functions and jumble them in semi-recognizable order.

[u/jicty]

But swearing is the Linux way, Just ask Linus.

[u/megaminxwin]

FUCK