05 - Literate Statistical Programming

This lecture, as the rest of the course, is adapted from the version Stephanie C. Hicks designed and maintained in 2021 and 2022. Check the recent changes to this file through the GitHub history.

Pre-lecture materials

Read ahead

Read ahead

Before class, you can prepare by reading the following materials:

1. https://rafalab.github.io/dsbook/reproducible-projects-with-rstudio-and-r-markdown.html
2. https://statsandr.com/blog/tips-and-tricks-in-rstudio-and-r-markdown

Acknowledgements

Material for this lecture was borrowed and adopted from

• https://rdpeng.github.io/Biostat776/lecture-literate-statistical-programming.html
• https://statsandr.com/blog/tips-and-tricks-in-rstudio-and-r-markdown

Learning objectives

Learning objectives

At the end of this lesson you will:

• Be able to define literate programming
• Recognize differences between available tools to for literate programming
• Know how to efficiently work within RStudio for efficient literate programming
• Create a R Markdown document

Introduction

One basic idea to make writing reproducible reports easier is what’s known as literate statistical programming (or sometimes called literate statistical practice). This comes from the idea of literate programming in the area of writing computer programs.

The idea is to think of a report or a publication as a stream of text and code.

• The text is readable by people and the code is readable by computers.

• The analysis is described in a series of text and code chunks.

• Each kind of code chunk will do something like load some data or compute some results.

• Each text chunk will relay something in a human readable language.

There might also be presentation code that formats tables and figures and there’s article text that explains what’s going on around all this code. This stream of text and code is a literate statistical program or a literate statistical analysis.

Weaving and Tangling

Literate programs by themselves are a bit difficult to work with, but they can be processed in two important ways.

Literate programs can be weaved to produce human readable documents like PDFs or HTML web pages, and they can tangled to produce machine-readable “documents”, or in other words, machine readable code.

The basic idea behind literate programming in order to generate the different kinds of output you might need, you only need a single source document—you can weave and tangle to get the rest.

In order to use a system like this you need a documentational language, that’s human readable, and you need a programming language that’s machine readable (or can be compiled/interpreted into something that’s machine readable).

rmarkdown

Today’s main choice for literate programming is to build documents based on Markdown language. A markdown file is a plain text file that is typically given the extension .md.. The rmarkdown R package takes a R Markdown file (.Rmd) and weaves together R code chunks like this:

```{r plot1, height=4, width=5, eval=FALSE, echo=TRUE}
data(airquality)
plot(airquality$Ozone ~ airquality$Wind)
```

Tip

The best resource for learning about R Markdown this by Yihui Xie, J. J. Allaire, and Garrett Grolemund:

• https://bookdown.org/yihui/rmarkdown

The R Markdown Cookbook by Yihui Xie, Christophe Dervieux, and Emily Riederer is really good too:

• https://bookdown.org/yihui/rmarkdown-cookbook

The authors of the 2nd book describe the motivation for the 2nd book as:

“However, we have received comments from our readers and publisher that it would be beneficial to provide more practical and relatively short examples to show the interesting and useful usage of R Markdown, because it can be daunting to find out how to achieve a certain task from the aforementioned reference book (put another way, that book is too dry to read). As a result, this cookbook was born.”

Because this is lecture is built in a .qmd file (which is very similar to a .Rmd file), let’s demonstrate how this work. I am going to change eval=FALSE to eval=TRUE.

data(airquality)
plot(airquality$Ozone ~ airquality$Wind)

Questions

1. Why do we not see the back ticks ``` anymore in the code chunk above that made the plot?
2. What do you think we should do if we want to have the code executed, but we want to hide the code that made it?

Before we leave this section, I find that there is quite a bit of terminology to understand the magic behind rmarkdown that can be confusing, so let’s break it down:

• Pandoc. Pandoc is a command line tool with no GUI that converts documents (e.g. from number of different markup formats to many other formats, such as .doc, .pdf etc). It is completely independent from R (but does come bundled with RStudio).
• Markdown (markup language). Markdown is a lightweight markup language with plain text formatting syntax designed so that it can be converted to HTML and many other formats. A markdown file is a plain text file that is typically given the extension .md. It is completely independent from R.
• markdown (R package). markdown is an R package which converts .md files into HTML. It is no longer recommended for use has been surpassed by rmarkdown (discussed below).
• R Markdown (markup language). R Markdown is an extension of the markdown syntax. R Markdown files are plain text files that typically have the file extension .Rmd.
• rmarkdown (R package). The R package rmarkdown is a library that uses pandoc to process and convert .Rmd files into a number of different formats. This core function is rmarkdown::render(). Note: this package only deals with the markdown language. If the input file is e.g. .Rhtml or .Rnw, then you need to use knitr prior to calling pandoc (see below).
• quarto: Similar to rmarkdown but can be used outside R. It is not an R package! Input files have the .qmd file extension.

Tip

Check out the R Markdown Quick Tour for more:

• https://rmarkdown.rstudio.com/authoring_quick_tour.html

Artwork by Allison Horst on RMarkdown

Create and Knit Your First R Markdown Document

When creating your first R Markdown document, in RStudio you can

1. Go to File > New File > R Markdown…

2. Feel free to edit the Title

3. Make sure to select “Default Output Format” to be HTML

4. Click “OK”. RStudio creates the R Markdown document and places some boilerplate text in there just so you can see how things are setup.

5. Click the “Knit” button (or go to File > Knit Document) to make sure you can create the HTML output

If you successfully knit your first R Markdown document, then congratulations!

Mission accomplished!

Websites and Books in R Markdown

Now that you are on the road to using R Markdown documents, it is important to know about other wonderful things you do with these documents. For example, let’s say you have multiple .Rmd documents that you want to put together into a website, blog, book, etc.

There are primarily two ways to build multiple .Rmd documents together:

1. blogdown for building websites
2. bookdown for authoring books

In this section, we briefly introduce both packages, but it’s worth mentioning that the rmarkdown package also has a built-in site generator to build websites.

blogdown

blogdown logo

[Source]

The blogdown R package is built on top of R Markdown, supports multi-page HTML output to write a blog post or a general page in an Rmd document, or a plain Markdown document.

• These source documents (e.g. .Rmd or .md) are built into a static website (i.e. a bunch of static HTML files, images and CSS files).
• Using this folder of files, it is very easy to publish it to any web server as a website.
• Also, it is easy to maintain because it is only a single folder.

Tip

For example, my personal website was built in blogdown:

• http://lcolladotor.github.io/ with source code at https://github.com/lcolladotor/lcolladotorsource

Other really great examples can be found here:

• https://awesome-blogdown.com

Other advantages include the content likely being reproducible, easier to maintain, and easy to convert pages to e.g. PDF or other formats in the future if you do not want to convert to HTML files.

Because it is based on the Markdown syntax, it is easy to write technical documents, including math equations, insert figures or tables with captions, cross-reference with figure or table numbers, add citations, and present theorems or proofs.

Here’s a video you can watch of someone making a blogdown website.

[Source on YouTube]

bookdown

book logo

[Source]

Similar to blogdown, the bookdown R package is built on top of R Markdown, but also offers features like multi-page HTML output, numbering and cross-referencing figures/tables/sections/equations, inserting parts/appendices, and imported the GitBook style (https://www.gitbook.com) to create elegant and appealing HTML book pages. Share

Tip

For example, the previous version of this course was built in bookdown:

• https://rdpeng.github.io/Biostat776

My team documentation website is also built with bookdown:

• https://lcolladotor.github.io/bioc_team_ds/

Another example is the Tidyverse Skills for Data Science book that the JHU Data Science Lab wrote. The github repo that contains all the .Rmd files can be found here.

• https://jhudatascience.org/tidyversecourse
• https://github.com/jhudsl/tidyversecourse

Note: Even though the word “book” is in “bookdown”, this package is not only for books. It really can be anything that consists of multiple .Rmd documents meant to be read in a linear sequence such as course dissertation/thesis, handouts, study notes, a software manual, a thesis, or even a diary.

• https://bookdown.org/yihui/rmarkdown/basics-examples.html#examples-books

quarto

quarto is newer and is described in further detail at https://quarto.org/. It’s what was was used for making this course website. As an alternative to using postcards for Project 0, you could use quarto instead following the instructions from https://quarto.org/docs/websites/website-blog.html. As examples check:

• https://github.com/lahuuki/lahuuki.github.io and https://lahuuki.github.io/

• https://github.com/daianna21/daianna_blog and https://daianna21.github.io/daianna_blog/

I’m super excited to share with you my personal blog. Here I’ll be posting about the math and statistics behind frequently used bioinformatic tools to analyze omics data, hopefully contributing to increasing our understanding around them.

Go check 👉🏼 https://t.co/W5PsRBCCYu pic.twitter.com/sRgcKFxsZy

— Daianna ((daianna_glez?)) July 21, 2024

• https://github.com/Bioconductor/biocblog and https://blog.bioconductor.org/

Tips and tricks in R Markdown in RStudio

Here are shortcuts and tips on efficiently using RStudio to improve how you write code.

Run code

If you want to run a code chunk:

command + Enter on Mac
Ctrl + Enter on Windows

Insert a comment in R and R Markdown

To insert a comment:

command + Shift + C on Mac
Ctrl + Shift + C on Windows

This shortcut can be used both for:

• R code when you want to comment your code. It will add a # at the beginning of the line
• for text in R Markdown. It will add <!-- and --> around the text

Note that if you want to comment more than one line, select all the lines you want to comment then use the shortcut. If you want to uncomment a comment, apply the same shortcut.

Knit a R Markdown document

You can knit R Markdown documents by using this shortcut:

command + Shift + K on Mac
Ctrl + Shift + K on Windows

Code snippets

Code snippets is usually a few characters long and is used as a shortcut to insert a common piece of code. You simply type a few characters then press Tab and it will complete your code with a larger code. Tab is then used again to navigate through the code where customization is required. For instance, if you type fun then press Tab, it will auto-complete the code with the required code to create a function:

name <- function(variables) {
  
}

Pressing Tab again will jump through the placeholders for you to edit it. So you can first edit the name of the function, then the variables and finally the code inside the function (try by yourself!).

There are many code snippets by default in RStudio. Here are the code snippets you might want to use:

• lib to call library()

library(package)

• mat to create a matrix

matrix(data, nrow = rows, ncol = cols)

• if, el, and ei to create conditional expressions such as if() {}, else {} and else if () {}

if (condition) {
    ## Case 1
} else if (condition) {
    ## Case 2
} else if (condition) {
    ## Case 3
}

• fun to create a function

name <- function(variables) {

}

• for to create for loops

for (variable in vector) {

}

• ts to insert a comment with the current date and time (useful if you have very long code and share it with others so they see when it has been edited)

# Tue Jan 21 20:20:14 2020 ------------------------------

You can see all default code snippets and add yours by clicking on Tools > Global Options… > Code (left sidebar) > Edit Snippets…

Ordered list in R Markdown

In R Markdown, when creating an ordered list such as this one:

1. Item 1
2. Item 2
3. Item 3

Instead of bothering with the numbers and typing

1. Item 1
2. Item 2
3. Item 3

you can simply type

1. Item 1
1. Item 2
1. Item 3

for the exact same result (try it yourself or check the code of this article!). This way you do not need to bother which number is next when creating a new item.

To go even further, any numeric will actually render the same result as long as the first item is the number you want to start from. For example, you could type:

1. Item 1
7. Item 2
3. Item 3

which renders

1. Item 1
2. Item 2
3. Item 3

However, I suggest always using the number you want to start from for all items because if you move one item at the top, the list will start with this new number. For instance, if we move 7. Item 2 from the previous list at the top, the list becomes:

7. Item 2
1. Item 1
3. Item 3

which incorrectly renders

7. Item 2
8. Item 1
9. Item 3

New code chunk in R Markdown

When editing R Markdown documents, you will need to insert a new R code chunk many times. The following shortcuts will make your life easier:

command + option + I on Mac
    (or command + alt + I depending on your keyboard)
    
Ctrl + ALT + I on Windows

Reformat code

A clear and readable code is always easier and faster to read (and look more professional when sharing it to collaborators). To automatically apply the most common coding guidelines such as white spaces, indents, etc., use:

cmd + Shift + A on Mac
Ctrl + Shift + A on Windows

So for example the following code which does not respect the guidelines (and which is not easy to read):

1+1
  for(i in 1:10){if(!i%%2){next}
print(i)
 }

becomes much more neat and readable:

1 + 1
for (i in 1:10) {
  if (!i %% 2) {
    next
  }
  print(i)
}

I also like to use the styler: Non-invasive pretty printing of R code package to automatically style my R code. I also use https://lcolladotor.github.io/biocthis/reference/bioc_style.html to style my code in a way that it is compatible with Bioconductor’s code standards.

## Install styler for automatically styling scripts
install.packages("styler")

## Install biocthis
if (!require("BiocManager", quietly = TRUE)) {
    install.packages("BiocManager")
}

BiocManager::install("biocthis")

## Example code for styling all files that end with .qmd
cat(readLines(here::here("scripts", "auto_style.R")))

styler::style_dir(     here::here(),     transformers = biocthis::bioc_style(),     filetype = "qmd" )

You might want to edit your ~/.Rprofile with `usethis::edit_r_profile()` and add this line to it:

## For the styler addin
# Affects the output of: styler:::get_addins_style_transformer_name()
# https://github.com/r-lib/styler/blob/acfb42acc2e558e7b57ef133f1470df78b5093fd/R/addins.R#L183
options("styler.addins_style_transformer" = "biocthis::bioc_style()")

For more information I have on my ~/.Rprofile file check https://lcolladotor.github.io/bioc_team_ds/config-files.html#rprofile.

RStudio addins

RStudio addins are extensions which provide a simple mechanism for executing advanced R functions from within RStudio. In simpler words, when executing an addin (by clicking a button in the Addins menu), the corresponding code is executed without you having to write the code. RStudio addins have the advantage that they allow you to execute complex and advanced code much more easily than if you would have to write it yourself.

Tip

For more information about RStudio addins, check out:

• https://rstudio.github.io/rstudioaddins
• https://statsandr.com/blog/tips-and-tricks-in-rstudio-and-r-markdown

Others

Similar to many other programs, you can also use:

• command + Shift + N on Mac and Ctrl + Shift + N on Windows to open a new R Script
• command + S on Mac and Ctrl + S on Windows to save your current script or R Markdown document

Check out Tools –> Keyboard Shortcuts Help to see a long list of these shortcuts.

Post-lecture materials

Final Questions

Here are some post-lecture questions to help you think about the material discussed.

Questions

1. What is literate programming?

2. Where can you find a list of other commands that help make your code writing more efficient in RStudio?

3. What type of output files can you make with rmarkdown?

4. If you make a website with R (html files), where would you host them for free?

5. What was the first literate statistical programming tool to weave together a statistical language (R) with a markup language (LaTeX)? (see former class material)

6. What is knitr and how is different than other literate statistical programming tools? (see former class material)

Additional Resources

Tip

• RMarkdown Tips and Tricks by Indrajeet Patil
• https://bookdown.org/yihui/rmarkdown
• https://bookdown.org/yihui/rmarkdown-cookbook

R session information

options(width = 120)
sessioninfo::session_info()

─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.1 (2024-06-14)
 os       macOS Sonoma 14.5
 system   aarch64, darwin20
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/New_York
 date     2024-08-28
 pandoc   3.2 @ /opt/homebrew/bin/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version date (UTC) lib source
 cli           3.6.3   2024-06-21 [1] CRAN (R 4.4.0)
 colorout    * 1.3-0.2 2024-05-03 [1] Github (jalvesaq/colorout@c6113a2)
 digest        0.6.36  2024-06-23 [1] CRAN (R 4.4.0)
 evaluate      0.24.0  2024-06-10 [1] CRAN (R 4.4.0)
 fastmap       1.2.0   2024-05-15 [1] CRAN (R 4.4.0)
 here          1.0.1   2020-12-13 [1] CRAN (R 4.4.0)
 htmltools     0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)
 htmlwidgets   1.6.4   2023-12-06 [1] CRAN (R 4.4.0)
 jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.4.0)
 knitr         1.48    2024-07-07 [1] CRAN (R 4.4.0)
 rlang         1.1.4   2024-06-04 [1] CRAN (R 4.4.0)
 rmarkdown     2.27    2024-05-17 [1] CRAN (R 4.4.0)
 rprojroot     2.0.4   2023-11-05 [1] CRAN (R 4.4.0)
 rstudioapi    0.16.0  2024-03-24 [1] CRAN (R 4.4.0)
 sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.4.0)
 xfun          0.46    2024-07-18 [1] CRAN (R 4.4.0)
 yaml          2.3.10  2024-07-26 [1] CRAN (R 4.4.0)

 [1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Former class material

This section contains older class material, that can be useful if you want to learn more about other valid R alternatives to the material we covered in class.

Sweave

One of the original literate programming systems in R that was designed to do this was called Sweave. Sweave enables users to combine R code with a documentation program called \(LaTeX\).

Sweave files ends a .Rnw and have R code weaved through the document:

<<plot1, height=4, width=5, eval=FALSE>>=
data(airquality)
plot(airquality$Ozone ~ airquality$Wind)
@

Once you have created your .Rnw file, Sweave will process the file, executing the R chunks and replacing them with output as appropriate before creating the PDF document.

It was originally developed by Fritz Leisch, who is a core member of R, and the code base is still maintained by R Core. The Sweave system comes with any installation of R.

There are many limitations to the original Sweave system.

• One of the limitations is that it is focused primarily on \(LaTeX\), which is not a documentation language that many people are familiar with.
• Therefore, it can be difficult to learn this type of markup language if you’re not already in a field that uses it regularly.
• Sweave also lacks a lot of features that people find useful like caching, and multiple plots per page and mixing programming languages.

Instead, folks have moved towards using something called knitr, which offers everything Sweave does, plus it extends it further.

• With Sweave, additional tools are required for advanced operations, whereas knitr supports more internally. We’ll discuss knitr below.

knitr

One of the alternative that has come up in recent times (it was first released in 2012; nowadays most people use rmarkdown directly) is something called knitr.

• The knitr package for R takes a lot of these ideas of literate programming and updates and improves upon them.
• knitr still uses R as its programming language, but it allows you to mix other programming languages in.
• You can also use a variety of documentation languages now, such as LaTeX, markdown and HTML.
• knitr was developed by Yihui Xie while he was a graduate student at Iowa State and it has become a very popular package for writing literate statistical programs.

Knitr takes a plain text document with embedded code, executes the code and ‘knits’ the results back into the document.

For for example, it converts

• An R Markdown (.Rmd) file into a standard markdown file (.md)
• An .Rnw (Sweave) file into to .tex format.
• An .Rhtml file into to .html.

The core function is knitr::knit() and by default this will look at the input document and try and guess what type it is e.g. Rnw, Rmd etc.

This core function performs three roles:

• A source parser, which looks at the input document and detects which parts are code that the user wants to be evaluated.
• A code evaluator, which evaluates this code
• An output renderer, which writes the results of evaluation back to the document in a format which is interpretable by the raw output type. For instance, if the input file is an .Rmd, the output render marks up the output of code evaluation in .md format.

Converting a Rmd file to many outputs using knitr and pandoc

[Source]

As seen in the figure above, from there pandoc is used to convert e.g. a .md file into many other types of file formats into a .html, etc.

So in summary:

“R Markdown stands on the shoulders of knitr and Pandoc. The former executes the computer code embedded in Markdown, and converts R Markdown to Markdown. The latter renders Markdown to the output format you want (such as PDF, HTML, Word, and so on).”

[Source]

distill

There is another great way to build blogs or websites using the distill for R Markdown.

• https://rstudio.github.io/distill

Distill for R Markdown combines the technical authoring features of the Distill web framework (optimized for scientific and technical communication) with R Markdown, enabling a fully reproducible workflow based on literate programming (Knuth 1984).

Distill articles include:

• Reader-friendly typography that adapts well to mobile devices.
• Features essential to technical writing like \(LaTeX\) math, citations, and footnotes.
• Flexible figure layout options (e.g. displaying figures at a larger width than the article text).
• Attractively rendered tables with optional support for pagination.
• Support for a wide variety of diagramming tools for illustrating concepts. The ability to incorporate JavaScript and D3-based interactive visualizations.
• A variety of ways to publish articles, including support for publishing sets of articles as a Distill website or as a Distill blog.

The course website from 2021 was built in Distill for R Markdown:

• Website: https://stephaniehicks.com/jhustatcomputing2021
• Github: https://github.com/stephaniehicks/jhustatcomputing2021

Some other cool things about distill is the use of footnotes and asides.

For example ¹. The number of the footnote will be automatically generated.

You can also optionally include notes in the gutter of the article (immediately to the right of the article text). To do this use the aside tag.

This content will appear in the gutter of the article.

You can also include figures in the gutter. Just enclose the code chunk which generates the figure in an aside tag.

References

Knuth, Donald E. 1984. “Literate Programming.” Comput. J. 27 (2): 97–111. https://doi.org/10.1093/comjnl/27.2.97.

Footnotes

1. This will become a hover-able footnote