Literate Programming, Programming, Emacs, Clojure

Literate [Clojure] Programming: Publishing Documentation In Multiple Formats

This blog post is the sixth, and the last, of a series of blog posts about Literate [Clojure] Programming in Org-mode where I explain how I develop my [Clojure] applications using literate programming concepts and principles.

This last post introduce a tool that leverage the side effect of having all the codes neatly discussed: it gives the possibility to automatically generate all different kinds of project documentation with a single key-binding. The documentation that we will generate are:

  1. Human readable HTML general project documentation
  2. Programming API documentation
  3. Book style complete project documentation in PDF

This series of blog posts about literate [Clojure] programming in Org-mode is composed of the following articles:

  1. Configuring Emacs for Org-mode
  2. Project folder structure
  3. Anatomy of a Org-mode file
  4. Tangling all project files
  5. Publishing documentation in multiple formats (this post)
  6. Unit Testing

Documentation, Documentation, Documentation!

Not all documentation is equal and depending on what you are looking for, different kinds and different formats of documentation may work better depending on a task. If I want to read in depth the architecture of a project, if I want to understand the underlying assumptions and decisions that lead to a certain design, if I want to read text with figures about data structures I would prefer reading article like documentation in form of a HTML page or a PDF document.

However, if I am programing something using a certain API, what I will be looking for is the documentation of the API with maybe some supporting material as required. I will be looking to understand how a specific function works, how its parameters are meant to be used, etc. I may also simply be looking for a consolidated list of available function calls. That kind of documentation is quite different than the previous one.

The beauty of Literate Programming is that as a software developed, I don’t have to write each of these types of documentation independently. I have everything I need in my literate programming files (in this case, in my Org-mode files) to generate all kinds of different purposes documentation.

Note that everything that is discussed in this series of blog posts has been applied to the org-mode-tests-utils project. I would strongly suggest you to browse that project along with its folder structure to see how this works in a real [tiny] project.

Now, let’s see how this can be accomplished.

publish.org

What we want to do to generate all the different kinds of documentation is simply having to open a Org-mode file, and to click C-c C-v t which will run all the code blocks in the file that will generate all the documentation we want. That file is called publish.org and is located in the /org/ root folder.

This file is split into three general sections:

  1. Generate HTML Documentation
  2. Generate API Documentation
  3. Generate PDF Documentation

Each of these sections contains the ELisp code blocks that are used to generate the different kind of documentation.

Generate HTML Documentation

What the HTML documentation generator does is to search and find [recursively] all the .org files that exists in the /org/ folder of your project. Then it will generate a sitemap.org file if it is not already existing that it will use to generate the HTML documentation.

Themes

Different themes and styles can be defined for the generated HTML pages. To enable a theme, you simply have to select the proper :html-head setting.

The main themes come from the org-html-themes extension. The theme currently being used is called readtheorg.

Publishing Options

A series of settings can be configured to create the documentation the way you want. Here are the main settings:

  • Settings
    • :base-directory
      • The base directory is the current directory which is the [project]/org/ directory where all the Org files are defined
    • :recursive
      • We specify that we want Org-mode to generate HTML files for each Org file in all children folder (recursively)
    • :publishing-directory
      • We specify where we want to publish the HTML documentation from the Org files
    • :publishing-function
      • We specify that we want to publish everything in HTML
    • :section-numbers
      • We don’t want any kind of section numbers generated by Org-mode
    • :with-toc
      • We want to include a table of content for each generated documentation file
    • :auto-sitemap
      • We want to generate a sitemap automatically. The file is named sitemap.html
    • :html-head
      • We want to specify a style sheet that will be used by each generated HTML file. It should be located in doc/html/css/

Additional settings and configurations are available from these two web pages:

Publish

To publish in HTML, you simply have to run the following code blocks:

(defun org-publish-org-sitemap-includes (project &optional sitemap-filename)
  "Create a sitemap of pages in set defined by PROJECT.
Optionally set the filename of the sitemap with SITEMAP-FILENAME.
Default for SITEMAP-FILENAME is `sitemap.org'."
  (let* ((project-plist (cdr project))
         (dir (file-name-as-directory
               (plist-get project-plist :base-directory)))
         (localdir (file-name-directory dir))
         (exclude-regexp (plist-get project-plist :exclude))
         (files (nreverse
                 (org-publish-get-base-files project exclude-regexp)))
         (sitemap-filename (concat dir (or sitemap-filename "sitemap.org")))
         (sitemap-title (or (plist-get project-plist :sitemap-title)
                            (concat "Sitemap for project " (car project))))
         (sitemap-sans-extension
          (plist-get project-plist :sitemap-sans-extension))
         (visiting (find-buffer-visiting sitemap-filename))
         file sitemap-buffer)
    (with-current-buffer
        (let ((org-inhibit-startup t))
          (setq sitemap-buffer
                (or visiting (find-file sitemap-filename))))
      (erase-buffer)
      (insert (concat "#+TITLE: " sitemap-title "\n\n"))
      (while (setq file (pop files))
        (let ((link (file-relative-name file dir))
              (oldlocal localdir))
          (when sitemap-sans-extension
            (setq link (file-name-sans-extension link)))
          ;; sitemap shouldn't list itself
          (unless (equal (file-truename sitemap-filename)
                         (file-truename file))     
            (let ((entry
                   (org-publish-format-file-entry
                    org-publish-sitemap-file-entry-format file project-plist)))
              (insert (concat "* " entry "\n"
                              "#+INCLUDE: " link "\n"))))))
      (save-buffer))
    (or visiting (kill-buffer sitemap-buffer))))
(setq org-publish-project-alist
      '(("org-mode-clj-tests-utils--doc-html"
         :base-directory "."
         :publishing-directory "../doc/html"
         :publishing-function org-html-publish-to-html
         :section-numbers nil
         :recursive t
         :exclude "fulldoc\\.org\\|project\\.org\\|tangle\\-all\\.org\\|setup\\.org\\|publish\\.org"
         :with-toc t
         :auto-sitemap t
         :sitemap-function org-publish-org-sitemap-includes

                                        ; ReadTheOrg Theme
         :html-head "<link rel=\"stylesheet\" type=\"text/css\" href=\"themes/styles/readtheorg/css/htmlize.css\"/>
<link rel=\"stylesheet\" type=\"text/css\" href=\"themes/styles/readtheorg/css/readtheorg.css\"/>
<script src=\"https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js\"></script>
<script src=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js\"></script>
<script type=\"text/javascript\" src=\"themes/styles/lib/js/jquery.stickytableheaders.js\"></script>
<script type=\"text/javascript\" src=\"themes/styles/readtheorg/js/readtheorg.js\"></script>")))
(setq org-publish-use-timestamps-flag nil)
(setq org-export-html-style-include-scripts nil
      org-export-html-style-include-default nil)

(org-publish-all)

Once everything is properly configured, (org-publish-all) is run and the HTML documentation get generated in the /doc/html/ folder. You can see what the generated HTML documentation looks like here.

Generate API Documentation

Programming API documentation is also easy to generate. In this case, what we are doing is to use the tangled code to generate the API documentation. Here we are using Clojure’s Codox API code generator to generate the documentation, but we could have used any other such libraries to do the same.

To generate the API documentation, we simply have to run this Clojure code block and use Codox’s API to generate the full documentation.

(use 'codox.main)

(generate-docs {:output-path "doc/api"})

The generated documentation will appear in the /doc/api/ folder. You can see what the generated API documentation looks like here.

Generate PDF Documentation

Finally, we can leverage Org-mode’s internal (org-publish-current-project) internal function to generate a PDF version of each Org-mode file within the /org/ folder (recursively). We leverage the org-latex-publish-to-pdf publishing function to generate the files in PDF.

(setq org-publish-project-alist
      '(("org-mode-clj-tests-utils--doc-pdf"
         :base-directory "."
         :publishing-directory "../doc/pdf"
         :publishing-function org-latex-publish-to-pdf
         :recursive t
         :section-numbers nil
         :with-toc t
         :auto-sitemap t)))
(org-publish-current-project)

The generated PDF documentation will appear in the /doc/pdf/ folder. You can see what the generated API documentation looks like here.

Conclusion

This is what conclude my series of six blog posts about how I do Literate Programming [in Clojure using Org-mode]. As I mentioned in another article, the problem of writing readable code which is well commented, well documented and well tested is that ideally we would have to focus on all these aspects at the same time, but given the development environments used by most people, it is not possible. You will plan an aspect of your program and write the code. Then if you are really lucky and you will find (or take) the time to write some documentation and create some unit tests. The problem is that each of these tasks are siloed: they are performed in isolation with 4 different states of minds, at 4 different times and hopefully within 4 weeks. The worse happens when you start fixing bugs or improving the code: comments, documentation and unit tests will often remain unchanged and lagging behind.

This is what Literate Programming is for me: a way to perform all these tasks at once, with the same state of mind, at the same time. This is a process to put in place, a new way to work. The problem is to put in place that process, that way to work, that enables you do to all this at once. I hope I have been able to put in place and explains a Literate Programming process that can work for you and shows the benefits of doing so.

I have the feeling that it will become more and more important to write readable code and to write about the thought process that lead to that written code. Much of the code we are writing in these days is code that manipulates and transform data, code that implement machine learning workflows and such. The kind of code that greatly benefit to be readable by many people other than the ones that write the code.

One thought on “Literate [Clojure] Programming: Publishing Documentation In Multiple Formats

  1. Hello Frederick,

    Thank you very much for the great guide
    it was great help and and gave me some great insights for
    starting with literate programming.
    It could be a great help with my studies.

    Regards

    Andreas

Leave a Reply to Nebucatnetzer Cancel reply