Consistent Technical Documents Using Emacs and Org Mode

hi there in this video I'm going to be demonstrating emacs's org-mode specifically how to use it with or babble to create technical documents like essays blog posts and how-to guides when writing about programming or other technical subjects you're often weaving blocks of source code program output and raw data in with your prose these supplementary materials are usually copied and pasted into your document from other sources which can be difficult and tedious to keep up to date as things change inconsistencies and errors can easily creep in when you hard code dynamic information like program output into your writing in this example we're going to be writing a blog post on how to manually create get objects using git plumbing commands on Python I chose this example because there's lots of dynamic content like strings and sha-1 hashes to contend with I'm using the GNU Emacs 26.3 and a terminal frame inside of tmux I also use GUI frames which can look pretty great in org with the same underlying server process which I'll show towards the end of the video if you don't know what org mode is you can think of it as an enhanced markdown language it's a plain text formatting syntax that is designed to be converted to a much fancier format like HTML or PDF you've probably used a markdown language maybe without even knowing it when creating readme files for github or gitlab or composed a message on an online discussion forum like reddit when we say org-mode we mean that Emacs is in a mode where it knows you're editing a file in the org markdown language this mode assists you by automatically rendering at your buffer as well as giving you some key commands to easily manipulate the structure of your document if you look at your Ord mode file outside of Emacs you can see what the plain text unrendered format looks like I mentioned earlier that markdown is designed to be converted to other formats let's see what it looks like when you export org to HTML ooh 1994 called they want their website back not to worry though you can add some of that good old web 2.0 stuff like CSS fonts and JavaScript I've set up the stuff in a file that will tell org to load in parse when exporting our document the setup file directive can reference a URL or a file this file has a bunch of HTML head directives that tell the HTML exporter to include local copies of JavaScript packages like jQuery and Twitter bootstrap as well as some CSS that implements this really cool "read the docs" inspired org theme called read the org by Fabrice Niessen now that we've configured the setup file let's regenerate the HTML uh yeah that looks a lot better now that that is sorted out let's get back to our essay in this section of the get essay we're writing we want to demonstrate how you can manually do the work of git init by creating the .git directory and the stuff that needs to go inside it org has this thing called org babel that lets you run blocks of code inside of your document you create these blocks with the begin source directive in this demo we're going to be using both shell and python blocks now when it comes to shell blocks it's a good idea to know some details about the environment that the shell code runs in things like your current working directory are sort of important I created a shell source block with a couple of commands in it and I told org to execute it org then wrote the output of those commands into my buffer below the source block by default your current working directory is going to be the same directory that the org file you're editing is in you can also see that when we exported our git.org file to HTML org wrote git.html to the same directory this directory is not going to work for us we want our code blocks to execute in a temporary directory that gets cleaned out every time we export the document we can add these things called header args to our source blocks to tell org details about how we want these blocks executed here we use the dir-header arg to point us at a scratch directory now when we execute this block it sets our directory to ~/org/tmp we don't want to have to remember to set this header arg on every source block though so we'll create a global property that applies to every source block in this buffer now when we reevaluate this block we're in the right directory even though we didn't explicitly set it as a header arg on the block remember when I said that we wanted to clean out the directory every time we export this document let's do that now we're going to tell org to include a file called code.inc where we'll stash some utility functions that we'll use later in this document this tag tells Emacs to load org-mode when it opens this file even though it doesn't end in .org the name directive above the beginning source block allows you to give the code block a name which will be necessary to refer to it later and this ":exports none" header arg tells org that we don't want to see this code block or its results show up in the export of our document but we do want it to execute because our git tutorial depends on starting without a .git directory then creating one and then adding an object to it we'll need to blow away the junk that previous exports of our documents right to the temp directory this way we start fresh this Col directive at the top of the file tells org to invoke the function that we just created to do that all right back to our git tutorial we're gonna run and get status to show that in the current state of the world git is not happy because it doesn't have a dot get dirt when we run git status here notice how we redirect standard error to standard out this is because org babel doesn't really capture standard error and if you want to demonstrate what an error message looks like this is what you have to do this is the part of the essay where we show how to initialize a git repo by hand I watched a YouTube video of this great talk that Tim Bergling gave back in 2013 called get from the bits up if you haven't seen it I'd highly recommend watching it my fake essay here shamelessly rips off his awesome talk now we create another shell source block where we create a bunch of directories add the head ref run tree to show the tree and then run git status to prove that git is happy both the treatment and the git status command add output to the result blocks if you were to try this for yourself you might be surprised to discover that in your version of Emacs only the very last line of the shell block is outputted this is because by default org is in value mode where the return value of the block you're executing is the result in my Emacs config I change this default to scripting mode where the results are anything written to standard out let's get rid of some old browser tabs and then export again to HTML to see what we're working with let me just add a quick note about what kit needs to be happy and now we're gonna move on to the next section of the document this is where we show how you can hash an arbitrary string and store it in the git object database now my first impulse is that that string should be hello world and then I thought maybe it should be something like welcome to seagull 2019 which is the conference I'm giving this talk ad and then I thought you know it should be dynamic so I can change it to whatever I want so let's do that now we're back in the code Inc file where we're going to keep all the little utility functions that we use throughout the document we're going to name this function hello and it's job is to just output whatever string we're using for our hash example to mix things up a bit we're gonna do this example in Python under the hood there's a Python 3 virtual environment that the script is running in and that end equals quote quote at the end of the print line is just to tell print not to output a trailing newline let's save it flip back to our document and then replace the hard-coded text hello world with an inline call to the hello function did it work let's export it to HTML and see what it looks like by default org renders the output of inline functions in mono space which I don't think looks very good here we can fix it by adding a header tag to our function invocation there we go much better now we're going to create a new source block and we're going to name it hash object this allows us to refer to the results of this block by name which we're gonna do later instead of echoing the literal hello world string I have what looks like a function invocation surrounded by a pair of double angle brackets this is what's called the no web method and when we execute it the results are now tagged by name here I paste in some verbiage about the get shot and at this point I want to mention that you don't always have to refer to the entire 40 character hash and that you can often use as few as four characters to uniquely identify it here I'm at a dilemma I want to hard-code o 5 d 5 into my document which are the first four characters of the hash in this example but if I change the hello string I change the hash and then my document is wrong if I forget to update all the hash references another inline function to the rescue will define a function named short hash and this time we'll add the VAR tag to specify that this function takes a variable named line as an argument and will provide a default because that's useful for testing if we hit ctrl C ctrl C to execute this block in the buffer we can see that it uses the default and it does indeed just output the first four characters of the line variable now we can just call this function inline and pass in the argument line equals hash object which is the result of the named shell block we created above now we export it to HTML and take a look and see what it looks like you'll notice that in this case having the result of the inline function show up in mono space is actually what we want alright now we're gonna take a look and see where get actually stashes the object on the filesystem and we'll create another shell block that does a tree to illustrate where that is at this point we're going to macro in some more verbiage okay now we're gonna create another shell block which we're calling my hello which is calling get cat file with - p4 pretty print on the short hash and it's gonna dump the object we stored in the database let's export to HTML and see what this looks like cool looking good okay for this next task we're gonna need to create a new function a utility function called hash - der it's gonna take as its argument a 40 character hash and it's going to output the relative path of where that hash is stored in the git object database so what we've done here is once again we've got a Python function with a variable called line and we've set a default to some hash so we can easily test this what the script does this is a Python 3f string that basically prints you know dot get slash object slash and then the first two characters of the hash followed by a slash followed by the remaining characters the are strip at the end just gets rid of any extra whitespace that may exist in the input and once again if we hit ctrl C ctrl C to execute it we can see that it does what we expect so here we are creating yet another shell block this time we're calling cat with the relative path of the object of a file system and we're piping it into the file utility and we're also doing the same thing and piping it into gun zip it's funny that the BSD file utility thinks this is a VAX executable but here we are okay here's another shell block that is calling hash tag air piping it into pigs E - D - decompress it piping that into hex tub so this will be useful in showing the format of a blob on disk when it's uncompressed we have it exported to HTML in a while so let's do that and you can kind of see that now with all this code in here it's taking longer to export looks good though okay here's some more verbiage that explains how the header works and I don't know if you can tell but I made a mistake I hard-coded the number of 22 which is the length of the string welcome to seagull 2019 hopefully this should be the last time we have to go back to code ink to create another little helper snippet or script so this one is called sterlin and it just prints the length of line and now we replace this hard-coded 22 with an in line called a sterlin and we should get the very same result in our out and we do sweep I mentioned at the start of the video that I'd also share the GUI version of Emacs so let me flip to it real quick and as you can see they look pretty similar except with one notable difference pretty snazzy I spend a lot of time in the terminal though and that's where I'm most comfortable someone to go back there okay now comes the fun part we're gonna write some code in Python that attempts to do what get hash object does so in the first part here we're gonna come up with the same sha-1 hash that get did for this blob object but we're gonna do it in Python we're typing this into an org mode buffer and it's not doing us any favors in terms of our Python coding so we're gonna hit control-c single quote to jump into a Python mode and now that we're in a Python mode you know we can type in some garbage and we see some error messages cool want to type in some more code here and you can see you you know it's gonna complete some function names for us and that's cool that's company mode in action I don't know why anybody would select this like that but that's what I did cool so you know Python environment this is L PI and you can just hit control C single quote to get back and then we just take control C twice and Bob's your uncle in this next section of the document we're going to delete the old object that was previously written with get hash object to the file system this code has some header args that we haven't used before the exports code are or to only export the code to the document and the results silent argh tells org not to write any results to our buffer if we hit ctrl C ctrl C to execute this block look mom no results okay so now we're on the final stretch of the document this is the last Python script that calculates the hash and writes it to disk which it just did now we verify that get cat file can read the file that we just wrote and spoiler alert it can so that brings us to the end of the document just writing some final thoughts here now this is the end of the document but not the end of the video I'm going to demonstrate how I publish some or documents next so we haven't exported here in a tick but I'm gonna get to that let's take a look at my Emacs initialization file here and I am NOT a great Emacs Lisp programmer I know just about enough Emacs Lisp to configure a team action that's about it alright now that that caveat is out of the way let me talk about this function my publish so the first thing it does is it turns on the table of contents and we want a table of contents when we export to HTML because that's that cool sidebar on the left with the table of contents that works well on HTML and then that function or HTML publish to HTML actually publishes to HTML the next thing we do is we turn off the table of contents because after that we're publishing to as key and we're publishing to gfm gfm stands for github flavored markdown and in both those formats the table of contents just looks like crap so that's why we're tearing it off so in a nutshell turn table of contents on publish HTML turn table of contents off publish as key and publish gfm okay so this section here this org published project a list section this is where you define your projects and I've defined a project called GUID I set its base directory which is the directory at this org file that we've been working on is in I defined a publishing directory which for me is like get slash public in my home directory we define a publishing function which I just showed you and we define a completion function and this is a list of functions they get called when publishing is done so let's check these functions out this function my Firefox basically just calls exec and execs Firefox and it sets the the URL for it to use as a file URL that points to my publishing directory and then this my get publish function this just executes a shell script that lives inside my published directory this shell script is just dumb simple all it does is commit the changes it uses the date in the commit message and then it pushes to github and get lab that's it now I'll show you what the publish workflow looks like so let's flip back to the document and hit the publish command and we select our project to publish and this is gonna take a little bit because it's running all the code blocks multiple times it's running it once when it HTML publishes once when it publishes Taz key and once when it publishes to github flavored markdown so let's see what time it is now it is 9:57 these files were pushed to get lab at 9:57 alright so let's take a look at the text file pretty cool I like how it puts little as key boxes around the code and the results that's pretty neat and let's take a look at some of the github flavored markdown and I gotta say this looks pretty good I often publish to gfm at work as an inch aviary step before finally exporting to confluence editing in Emacs is so much nicer than using the confluence editor punching yourself in the face is nicer than using the confluence editor but I digress anyway we are at the end of this video thanks for watching and I hope you enjoyed it and I hope you learned something if you want links to my dot files or source materials or any of that stuff you can find it all in the video's description catch you next time

Channel: spudlyo

Views: 39,251

Rating: 4.9888811 out of 5

Keywords: emacs, org mode, org babel, git

Id: 0g9BcZvQbXU

Length: 22min 43sec (1363 seconds)

Published: Sun Nov 17 2019

Reddit Comments

[u/ragoneio]

This is great, I learned a lot! Thank you for sharing.

[u/Sonnilon81]

This is an absolutely brilliant and very slickly produced video. Thank you so much.

I use org-mode a lot, but have still not yet fully learnt how to make use of its fantastic literate programming capabilities/org-babel. I'll watch/follow through this video again as it'll greatly speed up this learning process.

[u/demosthenex]

I'm impressed with the video. Can you share some details on your HTML settings? I haven't seen Org's html export look like that before.

[u/taudep]

More props to you on a great tutorial. You're well spoken and easy to follow. Keep up the good work.

[u/doolio_]

Really well produced and informative video. Well done.

[u/83bytes]

This is a brilliant video. Thank You for sharing.

[u/jghobbies]

TIL Some people pronounce it "babble"

Great job on the video, I use org-babel directly when I give talks (and to write mini tutorials to myself for things I'm likely to forget) but I've always relied on github's baked in org representation. Definitely going to be using bits from this, especially if you can export to confluence, which I've never actually tried.

Punching yourself in the face is better than using the confluence editor

Truer words have never been spoken.

[u/mcgruffle]

I was following this with my own sample and I have to say it's disconcerting removing files from a path given by a parameter. I do like that your init function removed specific types of files (rather than rm -rf or rm **).

I added the following to my initializing function to ensure that the current working directory was at least close to where it was expected.

if [[ $(pwd) == *"/go/src"* ]] # make sure we're in a go directory

I think with a lot of projects, the path doesn't matter, so I'd do everything in a temp directory and ensure that the the script was running underneath where I place my tmpdirs.

It's especially troubling because this function gets called on export of an include. If I fuck up the call parameters (which I totally did the first time I wrote it out), I want to make sure that it's not accidentally deleting files from somewhere important.

[u/[deleted]]

Very awesome demo of why Emacs is amazing. Thanks!

Please more advanced ways to use Emacs ;)