Clojure, Emacs, Literate Programming, Programming

Literate [Clojure] Programming Using Org-mode

Literate Programming is a great way to write computer software, particularly in fields like data science where data processing workflows are complex and often need much background information. I started to write about Literate Programming a few months ago, and now it is the time to formalize how I create Literate Programming applications.

This is the first post of a series of blog posts that will cover the full workflow. I will demonstrate how I do Literate Programming for developing a Clojure application, but exactly the same workflow would work for any other programming language supported by Org-mode (Python, R, etc.). The only thing that is required is to adapt the principles to the project structures in these other languages. The series of blog posts will cover:

  1. Project folder structure (this post)
  2. Anatomy of a Org-mode file
  3. Tangling all project files
  4. Publishing documentation in multiple formats
  5. Unit Testing

Clojure Project Folder Structure

The structure of a programming project can vary a lot. The structure I am using when developing Clojure is the one created by Leiningen which I use for creating and managing my Clojure projects. The structure of a simple project (in this case, the org-mode-clj-tests-utils project that I created for another blog post) looks like this:

- CHANGELOG.md
- LICENCE
- README.md
- resources
- pom.xml
- project.clj
- src
  - org_mode_clj_tests_utils
    - core.clj
- target
- test
  - org_mode_clj_tests_utils
    - core_test.clj

There are 4 main components to this structure:

  1. the project.clj file which is used by Leiningen to configure the project
  2. the src folder where the project’s code files [to be compiled] are located
  3. the target folder is where the compiled files will be available, and
  4. the test folder where the unit tests for the code sources are located

This kind of project outline is really simple and typical. Now let’s see what the structure would look like if this project would be created using Literate [Clojure] Programming.

Literate Clojure Folder Structure

The best way and cleanest way I found to create and manage the Org-mode files is to create a org directory at the same level as the src one. Then to replicate the same folder structure that exists in the src folder. The names of the source files should be the same except that they have the .org file extension. For example, the src/core.clj file would become org/core.org in the Org-mode folder, and the org/core.org file is used to tangle (create) the src/core.clj file.

The new structure would look like that:

- CHANGELOG.md
- LICENCE
- README.md
- resources
- org
  - project.org
  - org_mode_clj_tests_utils
    - core.org
- pom.xml
- project.clj
- src
  - org_mode_clj_tests_utils
    - core.clj
- target
- test
  - org_mode_clj_tests_utils
    - core_test.clj

The idea here is that all the files that needs to be modified related to the project would become a Org-mode file. Such files are the code source files, the test files, possible other documentation files and the project.clj file. When the Org-mode files will be tangled, then all the appropriate files, required by the Clojure project would be generated.

Anything I am writing for this project comes from a Org-mode file. All the development occurs in Org-mode. If someone would want to modify such a Literate Clojure application, then they would have to modify the Org-mode file and not the source files otherwise the changes would be overwritten by the next tangling operation.

Utilities Org-mode Files

Finally, I created a series of Org-mode files that are used to perform special tasks such as:

  1. Tangling all project files at once, and
  2. Publishing documentation in multiple formats

These are Org-mode files that can be executed to perform these tasks. In the case of tangling all project files at once, it would be necessary to use it if you haven’t changed the behavior of your Emacs to automatically tangle files on save.

The second file is to publish weaved documentation in multiple different formats (HTML, LaTeX, etc.) as required, all at once.

These two files are directly located into the /org/ folder. I will explain how they work in a subsequent post in that series. The final structure of a Literate Clojure project is:

- CHANGELOG.md
- LICENCE
- README.md
- resources
- org
  - project.org
  - publish.org
  - tangle-all.org
  - setup.org
  - org_mode_clj_tests_utils
    - core.org
- pom.xml
- project.clj
- src
  - org_mode_clj_tests_utils
    - core.clj
- target
- test
  - org_mode_clj_tests_utils
    - core_test.clj

Conclusion

As you can see, a Literate Clojure application is not much different. The way to program such an application is more profound than the small changes that occur at the level of the folder structure.

There is still an open question related to publishing this kind of Literate work on repositories such as Git: should only the org folder be added to a Git repository, or should we also add the files that get tangled as well? In an ideal World, only the org files would need to go into the repository. However, depending on the nature of work (work only accessible by you, or work accessible by a group of people that know Org-mode, or making the project public on GitHub, etc.) we may have to commit the tangled files too. In the case of an open source project, I think it is required since many people unfamiliar with Org-mode won’t be able to use the codebase because they won’t be able to tangle it from the Org files. For this specific reason, I tend to publish the org files along with all the files that get tangled from them. That way I am sure that even if the users of the library doesn’t know anything about Org-mode or Literate Programming they could still use the code. The only thing I try to take care of is to commit the Org file and the tangled file related to a specific change in the same commit, and I try not to create two commits, one for each file.

The next blog post of that series will explain how the Org-mode source files are actually created, what is their internal structure, how they are organized and used.

Clojure, Emacs, Literate Programming, Programming

Improving org-babel-clojure

In a previous blog post, I started to play with org-babel-clojure to improve its capabilities such that Clojure gets better integrated into Org-mode for creating notebooks and Literate programs. The first thing I wanted to do is to remove the 20 seconds timeout that was defaulted with the nrepl. That meant that it was not possible to run procedures for longer than 20 seconds before it died with a timeout. Once this was implemented, the next step was to add a new feature to see the underlying process of a code block. Because the nature of my work (extensive work with big datasets), my procedures take time to run (minutes… hours…) and much information [about the process] is output to the terminal. However, in the org-babel-clojure implementation, you had to wait until the code was executed before being able to see the processing. What I did at the time is to add a new :async code block parameter which told org-babel-clojure to output all the output of the nrepl, when it was being processed, in a new window.

That worked like a charm. However, after much interaction with Nicolas Goaziou, one of the core maintainers of Org-mode, it was clear that my implementation was not an asynchronous implementation but really just a live processing output.

At the same time, I did find another major irritant: if an exception was raised in my Clojure code, then nothing was output to Org-mode, it was simply silently dying. The only way to see the exception was to switch to the Clojure major mode (using C-c ') and to rerun the code block.

Here are the two new improvements to my org-babel-clojure implementation:

  1. Rename the :async block parameter to :show-process
  2. Output the exceptions and errors messages with the output and the value results parameter

By renaming to :show-process I remove the ambiguity of the feature. Eventually we should get to a real asynchronous process, but the issue is that it is much more complex than I initially thought and this is a problem being addressed in Org-mode for all backends and not just org-babel-clojure.

Then every exception or error messages returned by nrepl are appended the value or the output returned by the code block. That way, we immediately see that something is going wrong directly within Org-mode.

Here is the latest version of my org-mode-babel implementation:

(defvar nrepl-sync-request-timeout)

(defun org-babel-execute:clojure (body params)
  "Execute a block of Clojure code with Babel. The block can be executed
   synchenously by default or asynchronously with the :show-process parameter"
  (let ((expanded (org-babel-expand-body:clojure body params))
        (sbuffer "*Clojure Show Process Sub Buffer*")
        (show (if (assoc :show-process params) t nil))
        (response (cons 'dict nil))
        status
        result)
    (case org-babel-clojure-backend
      (cider
       (require 'cider)
       (let ((result-params (cdr (assoc :result-params params))))
         ; Check if the user want to run code asynchronously
         (when show
           ; Create a new window with the show output buffer
           (switch-to-buffer-other-window sbuffer)

           ; Run the Clojure code asynchronously in nREPL
           (nrepl-request:eval
            expanded 
            (lambda (resp) 
              (when (member "out" resp)
                ; Print the output of the nREPL in the asyn output buffer
                (princ (nrepl-dict-get resp "out") (get-buffer sbuffer)))
              (when (member "ex" resp)
                ; In case there is an exception, then add it to the output 
                ; buffer as well
                (princ (nrepl-dict-get resp "ex") (get-buffer sbuffer))
                (princ (nrepl-dict-get resp "root-ex") (get-buffer sbuffer)))
              (when (member "err" resp)
                ; In case there is an error, then add it to the output 
                ; buffer as well
                (princ (nrepl-dict-get resp "err") (get-buffer sbuffer)))
              (nrepl--merge response resp)
              ; Update the status of the nREPL output session
              (setq status (nrepl-dict-get response "status")))
            (cider-current-connection) 
            (cider-current-session))

           ; Wait until the nREPL code finished to be processed
           (while (not (member "done" status))
             (nrepl-dict-put response "status" (remove "need-input" status))
             (accept-process-output nil 0.01)
             (redisplay))

           ; Delete the show buffer & window when the processing is finalized
           (let ((wins (get-buffer-window-list sbuffer nil t)))
             (dolist (win wins)
               (delete-window win))
             (kill-buffer sbuffer))

           ; Put the output or the value in the result section of the code block
           (setq result (concat (nrepl-dict-get response 
                                                (if (or 
                                                      (member "output" result-params)
                                                      (member "pp" result-params))
                                                    "out"
                                                  "value"))
                                (nrepl-dict-get response "ex")
                                (nrepl-dict-get response "root-ex")
                                (nrepl-dict-get response "err"))))
         ; Check if user want to run code synchronously
         (when (not show)
           (setq response (let ((nrepl-sync-request-timeout 
                                 org-babel-clojure-sync-nrepl-timeout))
                            (nrepl-sync-request:eval
                             expanded (cider-current-connection) 
                                      (cider-current-session))))
           (setq result
                 (concat 
                  (nrepl-dict-get response (if (or (member "output" result-params)
                                                   (member "pp" result-params))
                                               "out"
                                             "value"))
                  (nrepl-dict-get response "ex")
                  (nrepl-dict-get response "root-ex")
                  (nrepl-dict-get response "err"))))))
       (slime
        (require 'slime)
        (with-temp-buffer
          (insert expanded)
          (setq result
                (slime-eval
                 `(swank:eval-and-grab-output
                   ,(buffer-substring-no-properties (point-min) (point-max)))
                 (cdr (assoc :package params)))))))
      (org-babel-result-cond (cdr (assoc :result-params params))
        result
        (condition-case nil (org-babel-script-escape result)
          (error result)))))
Clojure, Emacs, Programming

My Optimal GNU Emacs Settings for Developing Clojure (Revised)

It has been 2 years since I last revised my optimal GNU Emacs settings for developing Clojure. Since then, many things have changed in the Cider/Emacs ecosystem and it is why a revision of these settings is now warranted.

There is one set of settings that I will not discuss in this blog post, and these are all the settings related to Literate Programming using Org-mode. Since much needs to be said about these, they will be the topic of a subsequent blog post that will focus exclusively on that matter.

GNU Emacs

GNU Emacs is an awesome programming interface. It is said to be the programmable programming interface. It is terrific but it comes at a price: steep learning curve and much time spent testing and configuring useful packages. However, once these hurdles are passed, the sun starts to shine and a joyful journey begins.

I am personally developing on Windows 10, but these steps should be platform agnostic. You only have to download and install the latest GNU Emacs 24 version.

The first thing you have to do is to locate your .emacs file. All the configurations I am defining in this blog post goes into that file.

Packages

Once Emacs is installed, the first thing you have to do is to install all the packages that are required to develop in Clojure or that will make your life easier for handling the code. The packages that you have to install are:

  1. cider
    • Clojure Integrated Development Environment and REPL. This is the actual Clojure IDE. It includes everything to work with Clojure in Emacs. It runs the REPL, it has a debugger, etc.
  2. company
    • In-buffer auto-completion framework. This is now the best integrated auto-complete for Cider.
  3. monokai-theme
    • This is a new theme that is pleasant to look at and that works pretty well with Clojure code.
  4. rainbow-delimiters
    • Highlight nested parenthesis, brackets, braces a different color at each depth – This is really handy to visually see where you are with your parenthesis. An essential to have (in my view).
  5. highlight-symbol
    • Highlight all occurrences of a symbol in a buffer. This is really handy to find occurrences of variables, or function calls, etc.
  6. heml-ag
    • High performance file system search function. This is essential to find all occurrences of a [regex] pattern within local files.

Before installing them, we have to tell Emacs which package repositories it has access to so as to find these packages. At the top of your .emacs file, put:

;; Define packages archives repositories
(require 'package)

(add-to-list 'package-archives
  '("melpa-stable" . "http://melpa-stable.milkbox.net/packages/"))

(add-to-list 'package-archives 
  '("marmalade" . "https://marmalade-repo.org/packages/"))  

;; Initialize all the ELPA packages (what is installed using the packages commands)    
(package-initialize)

If you are editing your .emacs file directly into Emacs, and you can re-evaluate the settings file using Emacs, then by moving the cursor at each top-level expression end (after closing parenthesis) and press C-x C-e. However, it may be faster just to close and restart Emacs to take the new settings into account. You can use any of these methods for the following set of settings changes.

Before changing any more settings, we will first install all the required packages using the following sequence of commands:

M-x package-install [RET] cider [RET]
M-x package-install [RET] company [RET]
M-x package-install [RET] monokai-theme [RET]
M-x package-install [RET] rainbow-delimiters]] [RET]
M-x package-install [RET] highlight-symbol [RET]
M-x package-install [RET] helm-ag [RET]

Additionally, you could have used M-x package-list-packages, then move your cursor in the buffer to the packages’ line. Then press i (for install) and once all the packages are selected, you could have press x (execute) to install all the packages all at once.

In the list of commands above, M-x is the “meta-key” normally bound to the left Alt key on your keyboard. So, M-x usually means Alt-x.

Now that all the packages are installed, let’s take a look at how we should configure them.

Installing Leiningen or Boot

An important piece of software to install is a build tool for Clojure. The two most popular choices are Leiningen and Boot.

Installing Platinum Searcher

For the helm-ag package, you will have to install the Platinum Searcher application. Installing it on Windows is quite simple. Just download the proper package, and extract the pt.ex file somewhere on your computer (let’s say in c:/platinum-searcher). We will configure it later.

Configuring Keyboard

If you are using an English/US keyboard, you can skip this section. Since I use a French Canadian layout (On an English/US Das Keyboard!), I had multiple issues to have my keys working since all the binding changed in Emacs. To solve this problem, I simply had to define that language configuration option. Then I had to start using the right Alt key of my keyboard to write my brackets, curly brackets, etc:

;; Enable a Canadian French keyboard layout
(require 'iso-transl)

Configure Text Editor

There are a few settings we have to specify related to the Emacs text editor. We want to send the comment columns of Clojure comments to column 70 and we want to force the usage of UTF-8 as the default encoding. Finally we want to start emacs maximized in Windows:

;; UTF-8 as default encoding
(set-language-environment "UTF-8")

;; Set the default comment column to 70
(setq-default comment-column 70)

;; Every time a window is started, make sure it get maximized
(add-to-list 'default-frame-alist '(fullscreen . maximized))

Configuring Fonts

There is a new wonderful coding font that I discovered recently called the Input (Font for Code). This is a really neat font that works particularly well. You just have to go to their site, define the characteristics you want for it, download and install it locally.

Once it is installed on your system, configure it that way in .emacs:

;; Use the Input font size 12
(set-default-font "Input-12")

Cider

Now that the general Emacs settings are configured, let’s focus on Cider. We want to make sure that we enter in cider mode when entering in Clojure major mode. Then we have to specify that we want to use company-mode as the auto-completion framework and finally we want to specify that when we enter via the client in Cider, that we want a new line and to indent that new line according to where we are in the code.

;; Enter cider mode when entering the clojure major mode
(add-hook 'clojure-mode-hook 'cider-mode)

;; Turn on auto-completion with Company-Mode
(global-company-mode)
(add-hook 'cider-repl-mode-hook #'company-mode)
(add-hook 'cider-mode-hook #'company-mode)

;; Replace return key with newline-and-indent when in cider mode.
(add-hook 'cider-mode-hook '(lambda () (local-set-key (kbd "RET") 'newline-and-indent)))

Show Parenthesis Mode

Another handy feature is to enable, by default, the show-parent-mode configuration option. That way, every time the cursor points to a parenthesis, the parent parenthesis will be highlighted into the user interface. This is an essential must-have with Par Edit:

;; Show parenthesis mode
(show-paren-mode 1)

Rainbow Delimiters

Another essential package to have helps you out in maintaining these parenthesis. The rainbow delimiters will change the color of the parenthesis depending on how “deep” they are into the structure. Another essential visual cue:

;; rainbow delimiters
(add-hook 'prog-mode-hook #'rainbow-delimiters-mode)

Monokai Theme

The Monokai theme is joyful and really fun to work Clojure code with. This is the current theme I use.

;; Set theme & font size
(add-to-list 'custom-theme-load-path "~/.emacs.d/lib/monokai-theme")
(load-theme 'monokai t)

There is another configuration we need to do which is to make sure that the theme’s colors are used by company-mode such that the popup contextual menus are properly themed:

;;
;; Update the color of the company-mode context menu to fit the Monokai theme
;; @source: https://github.com/search?q=deftheme+company-tooltip&type=Code
;;
(deftheme monokai-overrides)

(let ((class '((class color) (min-colors 257)))
      (terminal-class '((class color) (min-colors 89))))

  (custom-theme-set-faces
   'monokai-overrides

   ;; Linum and mode-line improvements (only in sRGB).
   `(linum
     ((,class :foreground "#75715E"
              :background "#49483E")))
   `(mode-line-inactive
     ((,class (:box (:line-width 1 :color "#2c2d26" :style nil)
                    :background "#2c2d26"))))

   ;; Custom region colouring.
   `(region
     ((,class :foreground "#75715E"
              :background "#49483E")
      (,terminal-class :foreground "#1B1E1C"
                       :background "#8B8878")))

   ;; Additional modes
   ;; Company tweaks.
   `(company-tooltip-common
     ((t :foreground "#F8F8F0"
         :background "#474747"
         :underline t)))

   `(company-template-field
     ((t :inherit company-tooltip
         :foreground "#C2A1FF")))

   `(company-tooltip-selection
     ((t :background "#349B8D"
         :foreground "#BBF7EF")))

   `(company-tooltip-common-selection
     ((t :foreground "#F8F8F0"
         :background "#474747"
         :underline t)))

   `(company-scrollbar-fg
     ((t :background "#BBF7EF")))

   `(company-tooltip-annotation
     ((t :inherit company-tooltip
         :foreground "#C2A1FF")))

   ;; Popup menu tweaks.
   `(popup-menu-face
     ((t :foreground "#A1EFE4"
         :background "#49483E")))

   `(popup-menu-selection-face
     ((t :background "#349B8D"
         :foreground "#BBF7EF")))

   ;; Circe
   `(circe-prompt-face
     ((t (:foreground "#C2A1FF" :weight bold))))

   `(circe-server-face
     ((t (:foreground "#75715E"))))

   `(circe-highlight-nick-face
     ((t (:foreground "#AE81FF" :weight bold))))

   `(circe-my-message-face
     ((t (:foreground "#E6DB74"))))

   `(circe-originator-face
     ((t (:weight bold))))))

helm-ag

It is essential to have a local file system search functionality. This is what helm-ag provides. I configured it to use the Platinum Searcher. I also bound it to the key M-s for easy invocation:

;; Configure helm-ag
;; Make sure to have Platinum Searcher installed: https://github.com/monochromegane/the_platinum_searcher

(custom-set-variables
 '(helm-ag-base-command "C:/platinum-searcher/pt -e --nocolor --nogroup"))

(global-set-key (kbd "M-s") 'helm-do-ag)

Syntax Highlighting

It is often handy to be able to highlight symbols within a buffer. I use highlight-symbol for that purpose but I re-mapped its functionalities to keys more meaningful and natural to me:

; Syntax Highlighting
(require 'highlight-symbol)
(global-set-key (kbd "C-é") 'highlight-symbol-at-point)
(global-set-key (kbd "C-.") 'highlight-symbol-next)
(global-set-key (kbd "C-,") 'highlight-symbol-prev)
(global-set-key (kbd "C-;") 'highlight-symbol-query-replace)

Binding Some Keys

I like to bind some behaviors to the F-keys. What I want is to be able to run Cider and to switch frames (windows within monitors) in a single click. I also added a shortkey for starting speedbar for the current buffer; it is an essential for managing project files. What I did is to bind these behaviors to these keys:

(global-set-key [f9] 'cider-jack-in)
(global-set-key [apps] 'other-frame)
(global-set-key [f11] 'speedbar)

Fixing the Scroll

There is one thing that I really didn’t like, and it was the default behavior of the scrolling of Emacs on Windows. After some searching, I found the following configurations that I could fix to have a smoother scrolling behavior on Windows:

;; scroll one line at a time (less "jumpy" than defaults)
(setq mouse-wheel-scroll-amount '(1 ((shift) . 1))) ;; one line at a time
(setq mouse-wheel-progressive-speed nil) ;; don't accelerate scrolling
(setq mouse-wheel-follow-mouse 't) ;; scroll window under mouse
(setq scroll-step 1) ;; keyboard scroll one line at a time

Conclusion

The Emacs/Clojure ecosystem has improved quite a lot in the last two years thanks to Cider and the decision they made in its development. The configuration of Cider is now much simpler with company-mode. This blog post provides the latest improvement I did to my setup.

As I said in my introduction, there is a whole bunch of settings that I did not include in this blog post related to configuring org-mode for doing Literate Programming in Clojure. I will discuss these in its own blog post, likely to be published within the next week.