Keynote: Typesetting with Python - Brandon Rhodes - PyLondinium19

okay we're gonna start it's already helped us so it is my pleasure to introduce the last but not least keynote speaker of Pi Londinium 2019 we have Brandon Rhodes with us here he is coming all the way from Ohio which we are very grateful for thank you very much he is a Python programmer he was a pike on us chair and I don't want to say you're more of your time because I know you have lots of interesting things to say so over to your venting thank you and I want to thank the organizers I want to thank the organizers very much for inviting me it's been at least half a decade No yeah half a decade been been five years or more since I first got to visit London and I've really enjoyed my visit and I've really enjoyed the energy of the conference and especially how compared to other conferences I've been to in Britain how many different places you are all from I've enjoyed meeting you and I'm here to talk about typesetting with Python when I was growing up there was a book on my father's bookshelf he was an electrical engineer it attracted me because of the interesting design of its cover which showed an ancient way of trying to design the letter s it was by someone had never heard of named Donald e Knuth and it was called heck and meta font not text and meta font no he named a typesetting system that he wrote after the Greek word techne they had the same word in Greek for both a craft and an art and so he named his typesetting system not with the letter X but with the Greek letter Chi as its third letter Tek it introduced me as I read through the book and he explained it's working introduced me to the idea of a markup language I'd never heard of such a thing instead of being in an a WYSIWYG editor you just wrote plain text that you were completely in control of and then tech read the plain text and turned it into a document you had to learn some conventions some commands that started with back slashes and the proper use of curly braces but you got to write plain text without thinking much about how it would be laid out on the page and the text system swung into action and made it look like I can professionally typeset paragraph from a book the books lessons on how to type these dot tech input files were themselves a small course in typography you learned where to put non-breaking spaces because it turns out it is a minor typographic disaster if a title like doctor or mr. or mrs. ends one line of a paragraph separated from the name on the next line so I learned about non-breaking spaces I learned that these four pieces of punctuation are completely different and it is entirely wrong to use any of them for the others purpose the - the end - the M - and the minus sign look different in the case of the minus sign are at a different height on the line and have completely different purposes you will be frequently annoyed for the rest of your life if you learn the difference and again it's markup language even though all you had was ASCII keys back in the 70s there was a way to separately type each of those symbols into your input file its specialty was math typesetting Donald Knuth was a mathematician and that was the real reason for inventing Tec when math journals stopped paying for professionals to expensively set type by hand math papers looked so ugly that Knuth could no longer publish to survive as a mathematician he had to remedy the situation so that he could stand seeing his equations in print and so he took an entire year off I believe it was 78 might have been 77 to invent Tec and so it's specialty is inside of dollar signs you have a little language that kicks in underscore a subscript caret is superscript where you can just say I'll take the sum from k equals zero to infinity and out comes beautifully typeset math that looked to me like something from an expensive professional math book it was a beautiful system Peregrine he did not ignore non math problems like breaking paragraphs into lines Tec represents the words of a paragraph as fixed-width fixed with boxes separated by stretchy glue because to lay out a paragraph you try to make the lines the same length by stretching or shrinking those spaces this internal representation of the problem of breaking a paragraph into lines well it has a combinatorial problem a paragraph within positions at which text could be split into lines can be laid out two to the in different ways depending on whether you take each break or not how could we ever find the optimum way out I'll bet you thought dynamic programming existed only to be a whiteboard problem in interviews Donald Knuth actually found dynamic programming could solve that problem and provide the optimal solution for breaking each paragraph in two lines in an in squared worse case actually usually only on the order of the number of possible breaks so you put in plain text it turned it into boxes and glue and it managed to beautifully line up that right hand side of each paragraph just like in a book the output of Tec was beautiful but as I then someday I eventually had a computer powerful powerful enough to run it I found it was difficult to control once you set up the parameters layout proceeded largely outside of your control it reminded me of a problem that I've seen recently which is the tractor and trailer problem tractor has a motoring can move forward and back and the trailers are simply passive if you've ever tried backing up a single trailer it can be difficult they often today consider for factory floors and things like that two three four trailers if you've ever tried to back up a small toy train you might have an idea of the difficulty here backing up a trailer a tractor in trailers is today an open problem in AI you can find recent papers that talk about could we back up three trial trailers with an appropriate AI trailers are difficult to back up because the input the motion of the tractor has an increasingly distant relationship to the motion of the infant trailer and that problem trying to control tech sometimes felt similar and in general I would say that frameworks in programming and I use that term in its fully preoperative sense frameworks often have the same problem that you feel like there's this little opinion you've been asked for and it can be very hard to get the form that they forms library outputs to respond to the settings that you're trying to change controlling tech felt similar so I had an idea what if instead of typesetting systems that we merely configure and then run off and do their thing what if there were a typesetting library that left the programmer in control for difficult situations that you might want to customize recently I realized that typesetting and printing a book from Python was coming within technical reach there's now an industry called print on demand all you need to give them is a PDF and they can send you back a custom hardcover book that is all yours a real hardcover case found and Smith's own not something just put together with glue we now have technologies that largely replace all the things Donald Knuth had to invent on his own we now have TrueType and OpenType we don't have to invent the idea of vector fonts instead of a macro language there are many ways to represent paragraphs and headings and plain text I didn't even need to implement paragraph layout Andrew couch Lee translated that boxes and glue algorithm into Python long ago and called it Tech lib and we don't need to use Tex custom DVI output format PDFs are now the industry standard but I needed something to print this is the problem with little projects that you dream about and then I stumbled across a backup of my grandfather's essays which after he died in the 90s I'd backed up off of his computer I could write the typesetting myself in Python and have a gift to give to family members a hardback of his thoughts and memories and history so alright we're going to be re-implemented tech what would I do differently if I'm gonna try to do layout over again what's the thing I want to focus on I chose a specific first goal to see if I could not just to re-implement it but do something interestingly different what about different width columns as you're laying out the document that's not supported in tech it wants each column to be the same width as tech breaks a paragraph into lines it doesn't even know what page the paragraph will land on because first splits the paragraph into lines at some width that's a separate step from then figuring out how many lines of the paragraph fit on each page so you can't support graphic designs that have one width on the first page of your newspaper and maybe a different column width on the inside you can't support graphic designers that on the front page of the article have a single wide column to accommodate an image but then on the inside have a double column my idea was that the paragraph should ask for more space as it needs it so it learns about any width change when it asks for more space and has to cross to a new column I wanted to see if I could get that working because that would be a real advance over what tech had been able to do so my plan was to find a library for rendering PDF from Python and then invent a new page layout engine I decided to tackle them in that order I had already used to make a I think a program of a printed program for a conference I'd use report lab it's a PDF library in Python and so I hooked it up to Andrew couch Ling's Tech lib and I printed out the first paragraph of my grandfather's first essay about his great-grandfather in 1800s Alabama and the result as you can see was a disaster on what yes we're just going to have to stop right here this is the best that we can do and the interestingly enough when I use this font on the web I double checked it the word war looked fine report lab specifically as a library doesn't understand that TrueType and OpenType fonts come with a table a kerning table that tells you that some letters need to be pushed closer together or they'll leave a gap it didn't know about kerning so the word war is an unmitigated disaster here fortunately it turns out there was an alternative I'd always thought that the Qt library was kind of a useless way to write big bloated desktop applications but it turns out that desktop applications need a print button so QT has an entire PDF rendering engine inside of it but thank goodness reads the metrics of a font before putting it on the page alright so I have my PDF library I was moving again the input is going to be from however my text is stored a list of typesetting actions titles headings and paragraphs and now I simply need to design the routine that will run through them putting them on the page what API should the layout engine use to call each of these actions like heading and paragraph well let's start by asking what information doesn't actually need if we're gonna set it loose laying out content on page well I thought it would need to know the width of the text it's expected to layout will need to probably know the height of a column so I wrap that up into a little object and then it probably needs to know how much of the columns being used already it needs to know the y-coordinate that it should start at as it adds more content each time the paragraph needs another line it can look at the amount of letting it needs used to be a little sheet of lead that you put in between each line of type today it's just a little blank space to give a nice line height not crowd the text too much vertically and the fonts height and if the current y-coordinate plus the letting it needs to add plus the size of the font it needs to add is going to be more than the columns height we're out of room we asked for another column how do we ask for another column I had several possibilities I could try here maybe the column itself has a method called next maybe we passed a layout object that can give you new columns as you asked for them or I could just pass a next column callable number of different options I decided to pass a plain callable and I knew it was the right decision when I did it and that wouldn't have been true when I was a younger programmer I knew that if all the paragraph needs is a verb a next column that spits out new columns that's all that I should pass it why to avoid I decided I needed a name for it premature object-orientation there is an old saying in computer scientist computer science premature optimization is the root of all evil who said that donald knuth i didn't know that till i looked it up premature object-orientation which is my phrase is attaching a verb gratuitous lee to a noun when you don't actually need to yet the symptom is when you see code passing an object on which a function will only ever call a single method this is the symptom you're given an object why all you ever do is call that single method why not just pass the verb instead premature object orientation couples code aids only a verb to all the implementation details of the noun because you receive the whole thing and so because of some restraint that I've learned over my career I simply passed the collarbone next column that generates new space in the document I now had a rough plan for an actions inputs alright what now will in action return can for example a paragraph simply go ahead and draw on the output device no it turns out it cannot because there are a number of problems which I can illustrate by talking about headings heading is supposed to sit atop the content of widgets the head yes that's why it's called a heading it looks sort of like this this is a paragraph laid out by my Lander couch Ling's Tech lib library called by mine and paragraph paragraph ends and we have just enough room on this page for the heading and a line of text what if there's no room beneath a heading what if the column were a little smaller well then you get a typographic disaster if the column weren't high enough for that line the heading would stand by itself not heading anything the heading in that case is going to need to move itself to the next column far better to have a little blank space at the bottom of a column than to have a heading that doesn't have any content beneath it now you might think that that's going to be easy to avoid can the headings simply check whether there is room for an extra line beneath it but checking for a free line won't alas always work why because a paragraph might not choose to use the final line of a column because of something paragraphs do on their own they're tricky beasts called widow and orphan detection a single line paragraph might deign to remain at the bottom of the page and then our heading has something to head before the text moves to the next column but a multi-line paragraph will refuse to leave its opening line alone will refuse to leave it an orphan it will look and say I can't leave that line by itself that's considered a sin in typography so even if there's the space for a line a multi-line paragraph will pull it up to the top of the next page to look better leaving the heading stranded even though it looked like there was enough room for more content well okay if paragraphs are going to do tricky interior things decisions about where they land where how can the heading predict whether it will be stranded alone well it can either know everything about paragraphs and have inside of it all of the details about how paragraphs layout that doesn't sound wise or sustainable or I could invent a way for a heading to ask the next action to lay itself out speculatively and just kind of ask well if I stay here at the bottom of the page what paragraph will you do next but this is going to require undo the ability if layout goes wrong if I wind up stranded to back up because the headings going to have to add itself to the document and the following paragraph and if it worked if there are some if having done that there's some lines of text beneath a heading we're done if the heading finds that it wound up stranded and alone you're gonna have to undo the paragraph undo the heading start over again on the next Paige that means these routines cannot be writing to the PDF manually layout is going to need to return an intermediate data structure that the caller can inspect instead of just starting to write type all over our document consequence number two is that the intermediate data it needs to be easy to discard I went through several pencil and paper I don't even try writing code on at the stage of design I have tried to figure out what relationship would let me add stuff to a document plan that I could then just take away and I was startled to discover that a very elegant data structure for it was one I had never used in Python in Python I used a linked list I defined a line as not only belonging to a column having a y-coordinate and then having some graphics that if you decide to keep the line you draw on the page but I gave it a previous attribute which as I go down the page I said each lines dot previous to the line above it thus creating a linked list of lines that when I need to stop and make a decision I can run a paragraphs layout one way look at those lines run it another way and the two possible futures for my document automatically share this is storage all the previous document without my having to store it twice because they both share a reference to it a linked list lets us extend the document with any number of speculative layout speculative layouts which Python automatically disposes of as we discard them actions okay are now going to need a new argument because if they're going if their job the job of an in action is now to append new lines they need is an argument the most recently laid out line before the action goes to work ooh but look remember how a paragraph needs to know the column it's drawing in and the y position so far those are two attributes of the line itself I didn't know I'd be passing lines in as an argument but now that I know that this gets passed in I no longer need to separately pass in the column and the y-coordinate the paragraph learns everything about the current context merely by looking at this line object that I didn't even pass in for its own amusement I passed it in because it needs to append more lines but it wound up carrying almost all the information the paragraph needs in other words designing our return value wound up eliminating two of our input arguments always look for chances to simplify as you proceed with a design it's often pleasant how the how things simplify as you go along another nice thing is the symmetry that this concept of a line provides the line becomes a common currency that's both our argument we get one line passed in and our return value is we return a new terminal line back out all right but how will the heading action invoke the action that follows I haven't talked yet about how that heading talks to the paragraph and how will it tell the engine the following actions already laid out if the heading needs to lay out the paragraph it needs to tell the engine hey I've already laid out the paragraph don't do it over again now that I hand back control and return of value once again pencil and paper I looked at special callable and exception should these two things we co-wrote Eames that talk to each other I took a deep breath alright now I looked at the reality of however I arrange the conversation between the heading in the paragraph the real reality is that I just have a list of tuples of things that I need to put on the page and so I thought of something kind of radical what if I just passed the list into the heading and the current I know that sounds crazy you'd think I would hide it behind something but what if in this case I just tell the heading the truth hey there's a list of actions you're at position a go to town and when you return the concluding line from all the work you've done tell me how far to advance my counter before continuing incrementing and returning the a index what's an action invoke as many subsequent actions as it needs to to compare different outcomes so stepping back I wrote several routines to that spec I looked askance at the amount of repetition in my code that this created heading took actions in a section took actions in a both used them but a paragraph which doesn't care about the subsequent action also takes actions as nna as did other simple routines they just kept returning a plus-one because I laid myself out continued and seemed a bit verbose for opinionated actions my headings that care about what follows it's necessary to pass actions and a so that they can get to the next thing but simple actions that just lay themselves out and don't care about what comes next ignore them and so the simple actions at the bottom are receiving parameters they don't really need how can I eliminate actions in a from simple actions that don't need them this is a principle in programming called dry don't repeat yourself and when I thought of those words I suddenly heard the call of distant decades the 1990s called and said from the practices back in the 90s that all in Python I can introspect functions to learn if they take actions arguments or not so given a list of functions I can just leave actions off of these argument lists and use pythons introspection abilities to know whether to pass those arguments or not the 90s crazy times up then the early 2000s called I could have a special registry cuz functions or objects and can be enrolled in data structures I could have a registry for functions that don't need actions an a so I know to call them differently but late-2000s called I could have a decorator for functions that don't need actions in a that takes those two parameters but doesn't pass them to the function being wrapped all I would then need to do is put that decorator in front of each of my simpler routines and I wouldn't have those extra parameters don't repeat yourself so I looked at the different possibilities that Python makes possible and what did I decide I decided to repeat myself I decided after several minutes I'm probably about a half hour of deliberation trying different things out maybe a bourbon I made the radical and countercultural decision to simply past actions and eight every single routine why because of symmetry everything looks the same if I pass actions and a when I return to code months and years later I relearned my code by rereading it given a stack of functions that do exactly the same thing if half of them use one convention and half of them use another then I now have twice the number of conventions to relearn later and only half the number example of examples of each to relearn from so I would not have done this when I was younger I just chose verbose symmetry over a symmetric brevity which would always have been my choice earlier in my career as a reader I need routines that behave the same to look the same if I have a chance of being able to maintain my own software and so symmetry it remained I simply had all the routines follow the same API alright we're ready for a final design step I talked earlier about widows and orphans avoiding stranding the first or last paragraph of a line alone as a human with intelligence laying out type was done for hundreds of years in Europe how does that look in code well my paragraph routine was was a bit complicated because of widows and orphans a paragraph had to lay itself out if it stranded an orphan at the page bottom it had to go back and try all over again lay itself out if it there's been stranded a widow at the top of the next page had to try again inside of this if-else widow/orphan logic paragraph had a hidden simpler inner routine called repeatedly that did the actual paragraph layout but you couldn't get to it it was down at the bottom of all these ifs and else's so what if you had wanted to just call the simple paragraph layout routine well I had to start adding boolean options to the paragraph callable in order to turn on and off that outer level of if statements but boolean switches are often a hint that we have coupled what could actually be two different routines instead of stranding code inside of other code with a switch to turn it on and off can you pull it out separately composition when it's possible is always to be preferred over coupling and so if a heading it occurred to me can control things about the paragraph that follows it could I write and avoid widows and orphans routine that had no content of its own to offer to the page but that simply controlled the next item whatever it was to help it avoid the problem of stranding lines I thought through it avoiding an orphan easy here on the left a paragraph has stranded an orphan its first line at the bottom of a page all we need to do if a paragraph does that is have this previous action go ahead and use up with whitespace the bottom of the first page then let the paragraph layout it's very very easy to bump an orphan to the next page and fix it before calling the paragraph simply move to the next column what about avoiding a widow I tried to sketch out a solution and found it is nearly impossible why well because a widow is created when the very last line of the paragraph falls onto a separate Paige this means I have to lay out the paragraph all over again and it's next to the last line even though there's room for it in the column needs to decide to move to the next page how am I going in the middle of the paragraph making decisions to just convince it to not use the page like it's supposed to and move to the next column how would we ever convince a paragraph to move to the next column early I went and looked at the inside of the paragraph in each time the paragraph needs another line letting the height adds them together compares them to the column height and it's that trigger I need so much space and it's not available that convinces it to ask for another column how could we influence this choice well I could lie about the value of y I could create a line object that I pass it that has a Y that's a little bigger than it should be so the paragraph will run out of room early but then I'll have to go and fix all the line objects it returns to be back up where they belong on the page I could make a fake column object instead of the real column object it's a little shorter than normal and thus force the routine to move to the next page early we are looking desperately for parameters to tweak why because we're standing outside of the code that makes the decision so all of the ideas I was thinking of we're really bad ones it was it was that feeling of using a framework you're not quite in control of your thinking of what parameter can I possibly set here that will make the paragraph behave like I want we're on the outside is that really where we want to be during our codes crucial decision know when code is making a crucial decision you want to be in the room where it happens the room where it happens the room where it happens I want to be in the room where it happens right now the paragraph only consults us when it needs a whole new column right if it needs another page another column it calls the callback that we've provided it well what if I get this concept which actually was repeated all through my code of needing another line comparing and seeing if I have enough room and either returning a line in the same column or if we run out of room a line in the next column what if I wrap that up as a collar ball and so what if the paragraph calls back not only when it thinks it needs a next column what if instead of only calling me back when it thinks it's ready for the next column the paragraph calls back every time it needs a next line what if I get that crucial if statement I need to control that's down in the paragraphs own code and bump it up into a callable that I pass in then the widow/orphan logic can subvert the paragraphs normal decision simply by passing a custom next line function right inside of avoid widows and orphans I can just make a little wrapper and pass the wrapped version of next line to paragraph and I in complete control of exactly the point at which it decides to go ask for the next column and moves to the top of that page this design was a complete success and completely let me break out widow and orphan avoidance from paragraphs and turn it into a general routine which can apply to bulleted lists to tables to all kinds of information that I might want to apply the same logic to but did you catch why this approach is a success why why was I able to do this sleight of hand the fancy next line wrapper is so simple because we avoided premature object orientation what if instead of just passing next line we were passed a whole layout object and next line was merely a method how would you make and objects and already existing objects next line method returned a different value because you wanted it to when you monkey patch it would you wrap it in an adapter class would you implement the entire Gang of Four decorator pattern in object orientation customizing a verb one little verb can retry require trundling out an entire design pattern to wrap the object but if you pass colobus if you treat your verbs as first-class citizens which is one of the strengths of Python simple inline wrapper can put you in the room where it happens I've learned over the years and my designs to start verbose and don't be afraid of that and to simplify and collapse things down later I have come despite myself to value symmetry over special cases to avoid premature object orientation and to let verbs be first class citizens attach them to nouns only if there's a compelling reason like dynamic dispatch I plan if you want to look at some of this code parts of it work on release ink by typesetting Python library later this summer but you can already watch my progress on github among other things it laid out this entire presentation which is a PDF and so to conclude what was that book that I showed you a picture of earlier that is my grandfather's memoirs the story the family history going back to the 1820s his opinions on religion and politics and death all bound together in a hardback book sure print and design in Toronto Canada will print runs of as little as two books I wanted to hold it in my hand and proofread it and have a copy to give my father so we got every anything anything we needed to fixed before we handed it to other family members it'll be a great gift at people's birthday I hope and Christmas my grandfather's just the little essays he'd written and printed off on on printer paper and handed to family over the years when put all in one place ran to 328 hardcover pages we had no idea it amounted to that much material and so I was able recently to surprise my father I had not told him about this project by giving him a professionally printed hardback of his own father's writings that looked like a professionally typeset book even though as I admit it to him it was really time set with Python thank you very much [Applause] thank you very much huntin okay we have a few minutes for questions now in the audience anyone you know if you want to disagree with me on typesetting I'm delighted to thank you very much for this talk I'm delighted to hear it because I've been using tech for about twice as long as I've been using Python and this is probably the best attended talk on digital typography in the world this year and perhaps the year before the only thing I think that can rival it will be Don Knuth 80th birthday celebration which was surely well attended so my I'm sorry I don't have a question thank you Thank You Brendan that was really very lovely I have just a quite a banal technical question I'm very sorry when you received the original material that your grandfather wrote presumably it was written in something like word or something like that I guess in Roth oh okay so it already had good markup for I wouldn't the day's answer that you needed so this question was how will my grandfather's essay is marked up one or two of them were in a sort of a quirky a format most of them had been written on his UNIX PC in a format called in Roth which you can if you go look at the source code to any UNIX man page in Roth is the system of markup Bell Labs 1970s markup to mark perhaps and bold and italics and headings the main problem with it is that it's more a styling language than a markup language it typically doesn't say unless you're using the macros for a man page natively in Roth doesn't say here's a heading it just says Oh make these words bold and so I had to write a Python script that I had to kind of learn his conventions all right when there's a line by itself that's bold I will interpret it as a heading when there is a line with automatic indentation I will interpret it as formatted poetry because he included my great-great-great-grandfather who started a small sawmilling town in southern Alabama in his old age would set and write poetry we don't even know how he learned that there was such a thing and so I just had to learn which in Rauf command he tended to use before normal paragraphs and before poetry and so it has more special cases I think than any other code I've ever written any more questions yes so uh when will your grandfather's memoirs be in book stores I would need to find an agent and I would also need to make sure that everyone is dead because he wrote very candidly about what it was like growing up in the segregated south he talked very candidly about how different ethnicities within Birmingham different white ethnicities were divided and while I think that that everything he says is very friendly and well-meaning there are events he recounts that might embarrass sons and daughters of other family members or he had a developed strong friendships with other Boy Scouts and some of the portraits of them are of people more racist than the average person in Birmingham and in 1935 and so to publish the complete essays I would have to make sure that that no one was still alive who might be embarrassed by the way that their parents were portrayed or I guess I could try to anonymize those scenes but so some things are up on my website if you go look on the website from the 90s I've had some of his essays about the oldest history of Alabama up there and distant family members have sometimes contacted me because they learn the names of their great and great-great grandfather's from those essays but some of them will probably have to remain unpublished until I'm sure that that anyone who might be implicated or parents might be implicated either wanted them published or was gone so I guess the the next keynote is gonna be insuring everybody's dead with Python but my question is because we have to ask the important questions where all the - is the right dashes in Rauf did not give him the full palette of dashes and so as I as editor went through fixing spelling mistakes and rectifying things that needing coeur d'coeurs I assured that all of the dashes were correct dashes in some cases my Python script could just do it automatically if the things to each side of the dash are digits I turned it into an N - otherwise I had to know whether it needed to be a hyphen or an M - which I had to do manually this is gonna be the last question here last question so are you intended to just release it as a library or I do have ideas about how to plummet into existing markup languages I was planning so there is at the moment as far as I know no lingua franca between markup languages by which an implementation of markdown and an implementation of restructuredtext agree to express themselves in some neutral equivalent term heading paragraph paragraph so I think that it would need not just for markdown but for each particular markdown library because there's a you're probably I would choose a major one and it would probably just need a little bit of custom Python code that spoke to that particular markdown library because while I could choose to try to choose one format to represent them all and ask one of those markdown translators to move everything over to that generally there's impedance mismatching that can miss nuances so probably custom code to talk to markdown custom code to talk to restructure text might work best a good question thank you thank you all very much [Applause] you

Channel: PyLondinium

Views: 3,155

Rating: 5 out of 5

Keywords: PyLondinium, Conference, Python

Id: e7RVO0Sqr4s

Length: 46min 17sec (2777 seconds)

Published: Tue Jul 30 2019