Rob Hyndman | How Rmarkdown changed my life | RStudio (2020)

[Music] please join me in welcoming rob hindman you i'm rob hyneman i'm a professor at monash university in australia and i want to take you to a land far far away australia to a time long long ago when i look more like that actually we don't need to go that far back maybe that far back but actually i'm going to start about there so i'm 17 there so we're talking mid 80s and i had my first computer so the first time i was ever trying to type a document on a computer it looked a little bit like looked a little bit like this that's actually not my first word processor but it was a couple of years later it was called word star 2000 i thought it was awesome only one other person in the room old enough to remember word star 2000 unfortunately they went bust fairly quickly so i switched to word perfect a few more of them which was even more awesome because it was a markup language and you had two sections of the screen you could have wysiwyg at the top and down the bottom you had markup and you could see where bold was on and bold was off i mean even microsoft word does not have that today so this was good i used that for a while but then when i went to university and tried to type mathematics that wasn't so helpful so eventually i switched to using tech as in not latex tech um i used that for a while until somebody said hey there's this other thing just come out called latex which uh has sort of better markup facilities so i switched to latex and i used that for a long time many many years it was fantastic could write beautiful documents i wrote several theses using latex lots and lots of reports for clients paper academic papers and so on so that was for the writing side and then of course i was doing data analysis i'm a statistician so i needed something for the statistical side and around 1987 i discovered this new tool what was new to me was before the days of r before the days of s plus it was an early version of s running in a new ide that went bust in about a year but it was called ice and that was pretty cool so i was using s for a few years to do my statistical computing and using latex to do the writing so i haven't got a screenshot of s i couldn't find one but those of you who grew up knowing s we'll see recognize some of those books then eventually s plus came out and so s plus had quite a neat ide so s plus for the computing latex for the writing was pretty good actually you could do quite a lot except that of course you needed to keep these things in sync and i would be writing s-plus code generating graphics saving them as postscript graphics pulling them into my latex documents and if i change the s plus code i then have to rerun the latex document to make sure it was up to date and so i developed this way of doing that using um or there's i switched to r at some point too yeah about 2000 switched to r that's was the our website at the time at around the year 2000 hasn't changed very much has it but the only thing that's different here is it says down the bottom um where is it says somewhere has some sort of oh well here it is web page is currently under construction that that was there for a few years okay so i invented this way of trying to make these things work together so a make file is a thing used by programmers to compile programs and so you make a small change in some function you don't want to recompile everything so you can use that in other tools as well so i was using it for r and latex if i made a change in my r file it would recompile the graphics if i made a change in the latex file it would recompile the latex but it was smart enough to notice that if the graphics had changed and it needed to recompile the latex even if the latex file hadn't changed and i wrote these these sort of complicated make files to do this i thought this was super cool i thought everyone would do this i would i would change the way everyone did r and latex together i wrote blog posts about it um his and i tweeted about it um here's a tweet about it got one like so you know i had a following [Laughter] this was 2012. um but but this guy from the us called eway noticed and uh he tweeted back saying or another way of separating afrom reports and a little link to a thing he was developing called knitter and i i looked down and said yeah i know but i reckon my way simpler does nida have any real advantages over my approach with make files so i persisted on for a while and uh sort of kept an eye on on on what ui was doing and after a while i realized that you know he was right and that although makefiles are useful for a lot of things putting the r code and the latex stuff together was going to make a lot more sense and so i switched and i st and i started trying to write my academic papers using markdown but of course they had to have a certain look so then i ended up writing our markdown templates that's a recent paper i wrote in an our markdown template and i was writing paper client reports so i had to write on our markdown template for clients and this was life changing like this made my life so much easier to i was writing papers and writing reports um just had to have a couple of templates and everything was good so i put those things together and i i couldn't think of a decent name for my package so i called it monash ebs templates ebs being econometrics and business statistics the department in which i work and all you have to do is to use the template and you can generate these nice things and the there's before i show you that there's um you know there's lots of stuff that goes into the front end of an academic paper so we need a title we need a list of authors and where they come from their addresses the abstract keywords what number of working paper it is in my department they always want you to stick jel codes on things and so on so i just put all this stuff into the yaml and i can generate working papers i convinced some other people in the department that this was the way to go so if i put this in my yaml and it generates this beautiful front page and then the title page and then away we go with the report so it was sort of my first attempt at writing in a markdown template and as i said it was changed my life and then i thought well what else i do a lot of other writing maybe our markdown can be used for other things so at around the same time as this latex s-plus thing i also got into making websites the web was developed in 1993 i think it took about four months before i had a website and it looked a little bit like that that was in 1997 four years later but it didn't change that much in the first four years some academics still have websites that look like that mine evolved over time so eventually it looked like that this is handcrafted html because there was no other way to make websites you put it in a text editor you wrote html code um and uh look to something that looked like that oh somewhere here s plus functions for time series that became the forecast package for r eventually but the functions were originally on my website um still handwriting html code just getting better at it still more hand crafting of html code at this point i thought i should have my own domain name and so i bought one which is the one on the screen there don't go there i changed my domain name to something else with a.com on the end eventually didn't worry about the old domain name but because there were links to it all over the web from all of my papers unpleasant organization got it and has bad stuff on there so don't go there um then i thought handwriting html code is no fun there's these new things for content management systems so i switched to one called joomla that was painful so i switched to another one called wordpress which was a little better but slow and i was trying to write blog posts about r and every time i would put my r code in it would sort of strip out everything after the less than hyphen thinking that this is an html tag and then i'd have to go back in and sort of try to copy and paste my thing back in and i was ending up spending about as much time sort of trying to put all my stuff into the blog post as actually writing the code so that was no fun but this stage i switched to this url you're allowed to go to that url that's the current one change the theme for wordpress and then i thought this was so this brings me to about 2016 and ui wrote blog down so i thought okay i use our markdown for everything else from all my other writing let's see if i can make a website using this thing called blog down you need a theme i looked around the place i figured out that blogdan's built on hugo kieran healy had a great theme he had his website on hugo so i forked his website deleted his content stuck my website my content in with permission he says in his github repo that you're allowed to do that thank you kieran did all this on a very long flight between melbourne and amsterdam copying and pasting and making sure everything worked and then thought great now let's do the r bit and it didn't work unfortunately kieran's theme didn't work with blog down not every hugo theme works with blog down so fly it home again i start again this time i um found another theme and then i adapted it to look like kieran's because i still liked his website so my own blog down theme that's on that github repo and it's saved so much time doing a website that way okay so another thing i was doing was writing books i my first book published in the late 90s was this one it was by textbook standards an extremely successful book but i didn't make any money out of it because the publisher makes a great deal of money and they didn't give much to the authors and i decided that i would never ever do another book with a commercial publisher again because they screwed my students and didn't benefit me so i thought if i'm going to do another book i'm going to put it online for free because my students wouldn't buy those too expensive for the students to buy so they would be looking around on dodgy websites getting bad information about forecasting and using that instead of the book that was i thought was reasonably good so i thought let's do it online for free this was before book down existed i didn't know anything i mean you i hadn't thought about it as far as i know so i thought let's put it all i can do it this is and also before my website was on blog down so i've tried to make it in wordpress the first version of it came out in wordpress decided that wasn't so good so i switched and i just copied everything over to drupal in a long painful experience that worked better the first edition of the book came out in in drupal but then i saw a book down and i thought that's what i wanted but it's just a few years too late so the second edition of the book i did in book down and that meant that i could have a latex i could pave a pdf version and an online version and everything would work beautifully and it really did make an enormous difference okay so that's the uh that's the book as it is now that's the online version it's relatively simple css styling but it's just running book down the latex is more complicated because i wanted the latex to look a certain way that's a page from the latex version which is sold on amazon okay at this point i was getting right into our markdown i'd made websites i'd written books and i had templates to do things what about the slides that i use for talks so way back in the day my slides looked like that using latex seminar package and then after a while i switched to a package called prosper which gave nicer looking slides and then i switched to bema which was a little easier in some ways to do things and then i switched to our markdown and that is my slides that i use these days so that's a package called binb bin stands for bin is not beamer and it generates beamer output from our markdown and the bin bazon cran i'm using the monash theme which is why that has monash as a template on the front and it's very very easy this particular talk that is my yaml title author i switched the date out i'm not in the thing you can see the date's actually here but in the final version i switched the date out from my twitter handle change the font size pretty much that's all i had to do to the to the yama and away you go riding a mark down super good my cv which had been maintained for years i thought we could do this in our markdown too so there's my cv in our markdown that's using the vitae package written by mitch ohara wilde and me i called that particular template after my own name because that's the template i was using in latex but there's other templates that you can use and it's again the yaml is pretty simple you just stick in things like your twitter handle and github that creates this stuff at the top and then everything else is standard markdown there's other templates that we put in the package if you want other looking cvs okay my students were writing theses i said i found this thing called our markdown you should all be using our markdown it'll change your life i wrote them a nice template stuck it on my github account they have to fork it and then they just use that and it generates theses in exactly the format that the monash research office thinks is correct i became head of department recently the chair of my department i have to send memos to staff naturally what am i going to send my memos in the university template is a word document so i switched it out and made it into a markdown template and i send around memos like this with all the branding and so on i can take off the branding and i write the site i use exactly the same template for writing referee reports to for journals um really simple gamble in this case you've got a title you've got an author do you want the branding on or off so the monash branding at the top or not and then just whichever the name of the template which is monash ebs templates memo had to write letters now i've had a department with nice logos and stuff on the top again there was an awful word template that the university gave me so i turned it into a nice markdown template and put it in the package this time there's a little bit more in the yaml because you've got to say well i put in who i am what my qualifications are which come in under here um my address i made it general enough so that other people can switch their names in as well um who i'm writing to who their address is how i start what my signature file is and then it all just gets generated and it looks fantastic and the other thing i do as a professor as i write exams for students again the university gives you this awful word template whether the logo was actually the wrong aspect ratio it sort of squished and the fonts change halfway through the front page so i made that look better still looks like what mono says the template should look like and then the exam is all written in our markdown so it's reproducible the graphics you get pulled in everything's great so that's also in the package and all you have to do is to put in what you need on the front page which in my case is the unit code the title we have multiple campuses and you have to tick these boxes to about which campus the exam is for so you just put in a campus true and campus false and the box will get ticked in the template how long the exam is and so on and then away you go so basically everything i now write i write in our markdown i spend my life in a text editor either writing a markdown or writing yaml um or occasionally fixing up the packages that produce the templates what have i lost i can't actually remember how to write make files anymore um even though i thought it was the greatest thing i i just it's complicated and i can't remember if i have to have a make file i'll find one that did something similar and copy and paste it and hopefully that will work what have i gained and so much faster both in terms of the ability to write things but also the speed at which documents get rendered the speed at which my website runs that the speed at which i can update things it's just incredibly and it's an incredible gain in speed it's so much simpler the documents are really really simple i don't have all that latex preamble at the top of my documents i just have a fairly simple yaml header everything i do is text so it's really simple it's all github all on git repositories so you've got version control everything's reproducible there's enormous gains by switching to our markdown for pretty well everything in my life so when i was writing this i thought well do i write anything these days that's not in our markdown and i can only think of about about these items that i still type on my computer but i'm not inside on our markdown document one is i maintain the tennis club website one is uh i write shopping lists sometimes and i type them out and uh email social media tweeting and that sort of stuff so i thought well okay i can solve this so i went and recreated my tennis club website nr markdown before between the time i started this talk and now it's now all running in our markdown runs about 30 or 40 times faster than it used to run and it's much much easier to maintain i don't think i'll bother with the shopping lists they're actually not too bad without their markdown but email would be great so anyway i'm not sure what your plans are next but i'd like to see mail down please with a chrome browser plugin so when someone sends me an email i can just hit reply and it'll come up with an r markdown box then i can go knit and send and away so i just like to finally just say thank you to you i you really did make my life so much easier i'm sorry i didn't believe you when you first said have you looked at mirror but you were right and i was wrong thank you for that so the slides for this talk and all the links to packages and links to anything else that i've mentioned are going to be on that uh if you just go to that link there thank you perfect thank you we have time for just a few questions but i'll be sure to share all these with you any advice on converting awful word template to beautiful are markdown templates oh that's that's actually really hard i've i i look at the word template and i copy and paste the text and then i just start from scratch i don't try to do anything in the word styling itself i just start again because it's i find the word templating is too difficult to convert what which aspects of building templates are related yaml that are particularly challenging or error prone what advice would you give to a novice in this area um so almost all of the templating i've done has been with pdf so it's latex and the back end so you do have to be quite good at knowing about latex but i've been using it for 20 years so that was not a problem but writing a pdf template you have to know a reasonable amount of latex to know how to do that i have less experience at writing html templates and no experience at all of writing word templates so i don't i wouldn't even know how to do that yeah someone else asked do you use bootstrap or other frameworks to style your templates no okay and this is getting a few uh upvotes here i like our markdown but i find google docs much easier for collaboration and iteration with others any thoughts on that so google docs are fun as long as you don't have a lot of r to do but putting in our graphics inside a google doc is as you know there's cutting and pasting to happen there so that's painful so i find our markdown works much much better if you've got any real statistical computing happening and for collaboration purposes um you know i generally try and teach my collaborators how to use github um and mostly that that works okay occasionally i'm working with people that don't know github or or are not prepared to learn to use it and then it's a little painful yeah there's lots of emails and can you please change this section here's the words that i want to add in but and things like that yeah thank you very much rob

Channel: RStudio

Views: 9,006

Rating: 4.9450173 out of 5

Keywords: rstudio::conf(2020), Rob Hyndman, rstudio, data science, machine learning, python, stats, tidyverse, data visualization, data viz, ggplot, technology, coding, connect, server pro, shiny, rmarkdown, package manager, CRAN, interoperability, serious data science, dplyr, forcats, ggplot2, tibble, readr, stringr, tidyr, purrr, github, data wrangling, tidy data, odbc, rayshader, plumber, blogdown, gt, lazy evaluation, tidymodels, statistics, debugging, programming education, rstats, open source, OSS, reticulate

Id: _D-ux3MqGug

Length: 23min 18sec (1398 seconds)

Published: Tue Nov 24 2020