Software developers generally tells me that they don’t have to document most of anything since the code is the documentation: just read it. This is when I reply: the code tells me the how you did it (if lucky) the words around the code should tell me the why.

With the recent (last 15 years) emergence of Infrastructure as Code (IaC), new important specialized developer roles such as DevOps and now MLOps, I will argue that literate programming concepts are becoming more and more important to the software industry.

Infrastructure as Code

In the last fifteen years, we saw the emergence of several Domain Specific Languages (DSL) to help system administrators to manage and provision their infrastructure. Those DSL revolutionized the way infrastructures were created and cared for.

Infrastructures were now entirely defined in plain text files. Those files could be versioned, they would get special treatments in IDEs, complete infrastructure could be rollbacked, etc.

But, there is one special characteristics that IaC has: the side effects of the “infrastructure code” are huge because a single line can lead to provision, or destroy, huge number of hardware resources which can have dramatic physical or monetary impacts.

Importance of Documentation

In this context, I argue about the importance of literate programming principles to code and maintain IaC. Some of the DSL are very opaque, a very small change can have dramatic side effects. IaC also has deal with versions of several tens, if not hundred, pieces of software that have to work together. IaC creates very complex networks of computers in different regions of the World.

None of that is self evident in code written in those DSL. The why needs to be documented very carefully, and documentation needs to be as close as possible to the DSLs code because every time something changes in the infrastructure, the change needs to be reflected in the text that describes the rational of that piece of infrastructure. And finally, both the how and the why will be carefully peer reviewed in a pull request. 

Org-mode: Agnostic Literate Programming Framework

Considering that there exists a specific DSL per framework (Terraform, Ansible, Docker, Puppet, Chef, etc.) it is important to have a literate programming framework that is language agnostic (unlike CWeb, nbdev, etc.). This is why I strongly support Org-mode. If the DevOps/MLOps developers works within Emacs, they have all the power of all the major modes already existing in Emacs to manipulate the code of those DSL within the code blocks. If they don’t, they can always use their favorite IDE with a Org-mode command line utility like OrgWeb.

Few Examples

Let’s take a look at what it looks like in the wild. Here are two examples, one that describes a Dockerfile and a series of Ansible playbooks, properly rendered in GitHub.

Literate Dockerfile

The first example is orgweb’s Docker.org file where the Dockerfile if generated. As you can see, everything of importance about the generated Docker image is stated. The version of Alpine Linux used, which version of Emacs is shipped with, the reason why we choose Alpine in the first place, etc.

Then it explains why the Dockerfile needs to install the tf-dejavu package, and what happens if it is not installed. And then it explains why the install.el and site-start.el files are being copied over and what they are used for. And finally, why install.el is being run while building the image.

Literate Ansible

Here is another example from a project I stumbled upon recently. This repository is a set of Ansible playbook to provision a series of infrastructure resources such as a docker registry, longhorn, etc.