My partner Mike Bergman got me interested in notebook style programming user interfaces maybe a year ago. We wanted to find a way to easily experiment with different data management structures and frameworks we are developing at Structured Dynamics. The idea behind a Notebook was quite interesting: it is to run code snippets anywhere in a document, to see the results within that document and finally to be able to document the process. Then if something changed in the data, or in the code, then each code snipped within a Notebook could be rerun at any time, and the results updated. This is a great way to do experimentation, to keep tracks of the tests your are doing and to document the whole process.

The idea is really interesting for the kind of work we are doing. I tested the Gorilla REPL which is an implementation of this style user interface in Clojure. Other such interfaces exists in other programming languages like IPython, Wolfram, etc. However, I always had an issue with what I was using: I had a hard time re-purposing the content I was creating; I couldn’t easily export this information in different format (blog posts, papers, etc.). Saving, reloading, re-running in different environment was often too much trouble: until I find Org-mode.

I am not sure why I didn’t came across Org-mode before, maybe because it was not advertised as as “notebook style programming user interface” but this is really what it is (mostly) all about, at least to me. As far as I know, this is the only such software that let you work with any kind of programming language in the same notebook. It can also export the notebooks in virtually any formats (several formats are supported by Org-mode itself, others can be exported using Pandoc).

This being said, I started experimenting with Org-mode to create different kind of Notebooks using Clojure. I am using notebooks that shows how to use different APIs we are creating, or ones that shows how different data processing workflows actually works or that shows how some structures (like UMBEL) have been created and how they can be leveraged. I am also creating notebooks to research and experiment different kind of algorithms that we are trying to implement in our products, or to do bug investigation reports for our clients, or… the possibilities are probably endless. But the core idea is almost always the same: communication. We write these notebooks to communicate (write) information for other people to consume (or more important, his future self).

Given this kind of tasks that I am performing in a notebook, I often have to run procedures that may takes minutes or even hours before their processing is finalized. However, as you will see below, running procedures that takes minutes to finalize is a show stopper with the current Org-mode Org-babel-clojure (ob-clojure.el)= package that let Org-mode to run Clojure code.

Before outlining the issues I had with the current implementation of the Org-babel-clojure package, let me explain how I installed and configured Org-mode locally.

First of all I installed Org-mode contribs from ELPA, then I configured it that way in my .emacs file. Note that I made multiple little changes here and there to end-up with the kind of editor I am comfortable to use. So this is about installing, enabling and tweaking Org-mode in Emacs:

;; Configure Org-mode with Cider

;; Load Org-mode
(add-to-list 'load-path "~/.emacs.d/lib/org-mode/")
(require 'org)

;; Here I specify the languages I want to be able to use with Org-babel.
(org-babel-do-load-languages
 'org-babel-load-languages
 '((clojure . t)
   (sh . t)
   (emacs-lisp . t)))

;; Specify the Clojure back-end we want to use in Org-mode.
;; I personally use Cider, but one could specify Slime
(setq org-babel-clojure-backend 'cider)

;; Let's have pretty source code blocks
(setq org-edit-src-content-indentation 0
      org-src-tab-acts-natively t
      org-src-fontify-natively t
       org-confirm-babel-evaluate nil
      org-support-shift-select 'always)

;; Useful keybindings when using Clojure from Org
(org-defkey org-mode-map "\C-x\C-e" 'cider-eval-last-sexp)
(org-defkey org-mode-map "\C-c\C-d" 'cider-doc)      

(require 'cider)

;; Remove the markup characters, i.e., "/text/" becomes (italized) "text"
(setq org-hide-emphasis-markers t)

;; No timeout when executing calls on Cider via nrepl
(setq org-babel-clojure-nrepl-timeout nil)

;; Turn on visual-line-mode for Org-mode only
;; Note: you have to install "adaptive-wrap" from elpa
(add-hook 'org-mode-hook 'turn-on-visual-line-mode)

;; Enable Confluence export (or any other contributed export formats)
(require 'ox-confluence)

Note that most of these configurations comes from the Org-babel-clojure webpage.

The first issue I encountered is when I started to run code that was taking longer than 10 seconds. Every time I was running such code, I ran into the follow error:

“nrepl-send-sync-request: Sync nREPL request timed out”

What this error means is the the synchronous request to nREPL (the Clojure back-end that run the actual code written into Org-mode) timeout. I was really not expecting a query to timeout that way. This led me to start reading the Org-babel-clojure code to see where such an error may be coming from. However, I have to do a disclaimer here: I never really looked into Elisp code until now. The only other work I did with Elisp was to configure Emacs so be indulgent with me and report all awkward code I may be writing here.

My journey started with the ob-clojure.el which is the file used to make the bridge between Org-mode and the Clojure back-end (Cider/nREPL in this case). It is after reading that code that I noticed the following function: org-babel-execute:clojure which appeared to be the thing that is run when we run a Clojure code block in Org-mode. Then I noticed the call to the function nrepl-sync-request:eval. That needed to be the culprit and what sent this Sync timeout error. I found this function in the Cider code. But then I found this other function that is called by the later: nrepl-send-sync-request. It is when I read this function that I noticed the nrepl-sync-request-timeout variable. Looking back at org-babel-execute:clojure I couldn’t see where I could define this timeout parameter. I looks like it was not possible to define it, which was a big issue to me since I needed to be able to run procedure that takes minutes to run.

It is at that time that I choose to hack the ob-clojure.el code to expose that timeout setting such that I could setup it properly for my own needs. The code I created for that purpose is:

; Addition of the org-babel-clojure-nrepl-timeout setting
(defvar org-babel-clojure-nrepl-timeout nil)

(defun org-babel-execute:clojure (body params)
  "Execute a block of Clojure code with Babel."
  (let ((expanded (org-babel-expand-body:clojure body params))
        result)
    (case org-babel-clojure-backend
      (cider
       (require 'cider)
       (let ((result-params (cdr (assoc :result-params params))))
         (setq result
               (nrepl-dict-get
                ; Addition of the org-babel-clojure-nrepl-timeout setting
                (let ((nrepl-sync-request-timeout org-babel-clojure-nrepl-timeout))
                  (nrepl-sync-request:eval
                   expanded (cider-current-connection) (cider-current-session)))
                (if (or (member "output" result-params)
                        (member "pp" result-params))
                    "out"
                  "value")))))
      (slime
       (require 'slime)
       (with-temp-buffer
         (insert expanded)
         (setq result
               (slime-eval
                `(swank:eval-and-grab-output
                  ,(buffer-substring-no-properties (point-min) (point-max)))
                (cdr (assoc :package params)))))))
    (org-babel-result-cond (cdr (assoc :result-params params))
      result
      (condition-case nil (org-babel-script-escape result)
        (error result)))))

What I modified in this code is to add a new global setting org-babel-clojure-nrepl-timeout. If this setting is nil then there won’t be any timeout, otherwise the timeout value will be in seconds. What I did is simply to bind its value to the nREPL setting nrepl-sync-request-timeout and be done with it.

That solved this issue. After I updated ob-clojure.el accordingly, I could run Clojure code that may takes several minutes in Org-mode! That was fanstastic, but it was not optimal.

In fact, when I am running workflows that may take 30 minutes to finalize, I normally output processing steps in the REPL such that I know where the process is and what it is currently processing.

The problem with the current implementation of Org-babel-clojure is that it uses the synchronous API of the nREPL. What I want is to be able to run Clojure code asynchronously such that I can get some feedbacks (via the REPL) from the procedure I am running. This opened a kind of a Pandora box, and something that looked simple turned out to be more complex than anticipated for someone without any knowledge into Elisp, internal mechanisms and APIs of Emacs.

The next goal I had is to try to make Org-babel-clojure asynchrone. What I wanted is to be able to get, somehow, was the output of a Clojure procedure when that procedure was outputing something to the REPL. My second journey started after reading John Kitchin’s blog post about Asynchronously running Python code into Org-mode code blocks. What I found out is that Python code was run via a sub-process which run the Python interpreter. John’s solution was to use a local file to write what the interpreter is outputing and then to feed that output to a new window that got created by John’s function.

I took that example as a given, and then I tried to implement the same solution, but for Clojure (without knowing what I was really doing). It is in this process that I found that the Clojure solution to that problem would be quite different than John’s. There is an asynchronous API in nREPL, it is just that it is not used in Org-babel-clojure. What I ended-up using from John’s example is not his code, but his core idea: using a new window to output the asynchrone process and then to kill it once the processing is finalized and before populating #+RESULSTS section of the Org-mode file.

After much testing and debugging I ended-up with the following solution to my problem:

(defun org-babel-execute:clojure (body params)
  "Execute a block of Clojure code with Babel."
  (lexical-let* ((expanded (org-babel-expand-body:clojure body params))
                 ; name of the buffer that will receive the asyn output
                 (sbuffer "*Clojure Sub Buffer*")
                 ; determine if the :async option is specified for this block
                 (async (if (assoc :async params) t nil))
                 ; generate the full response from the REPL
                 (response (cons 'dict nil))
                 ; keep track of the status of the output in async mode
                 status
                 ; result to return to Babel
                 result)
    (case org-babel-clojure-backend
      (cider
       (require 'cider)
       (let ((result-params (cdr (assoc :result-params params))))
         ; Check if the user want to run code asynchronously
         (when async
           ; Create a new window with the async output buffer
           (switch-to-buffer-other-window sbuffer)

           ; Run the Clojure code asynchronously in nREPL
           (nrepl-request:eval
            expanded 
            (lambda (resp) 
              (when (member "out" resp)
                ; Print the output of the nREPL in the asyn output buffer
                (princ (nrepl-dict-get resp "out") (get-buffer sbuffer)))
              (nrepl--merge response resp)
              ; Update the status of the nREPL output session
              (setq status (nrepl-dict-get response "status")))
            (cider-current-connection) 
            (cider-current-session))

           ; Wait until the nREPL code finished to be processed
           (while (not (member "done" status))
             (nrepl-dict-put response "status" (remove "need-input" status))
             (accept-process-output nil 0.01)
             (redisplay))

           ; Delete the async buffer & window when the processing is finalized
           (let ((wins (get-buffer-window-list sbuffer nil t)))
             (dolist (win wins)
               (delete-window win))
             (kill-buffer sbuffer))

           ; Put the output or the value in the result section of the code block
           (setq result (nrepl-dict-get response 
                                        (if (or (member "output" result-params)
                                                (member "pp" result-params))
                                            "out"
                                          "value"))))
         ; Check if user want to run code synchronously
         (when (not async)
           (setq result
                 (nrepl-dict-get
                  (let ((nrepl-sync-request-timeout 
                         org-babel-clojure-nrepl-timeout))
                    (nrepl-sync-request:eval
                     expanded (cider-current-connection) (cider-current-session)))
                  (if (or (member "output" result-params)
                          (member "pp" result-params))
                      "out"
                    "value"))))))
      (slime
       (require 'slime)
       (with-temp-buffer
         (insert expanded)
         (setq result
               (slime-eval
                `(swank:eval-and-grab-output
                  ,(buffer-substring-no-properties (point-min) (point-max)))
                (cdr (assoc :package params)))))))
    (org-babel-result-cond (cdr (assoc :result-params params))
      result
      (condition-case nil (org-babel-script-escape result)
        (error result)))))

The first thing this code does, is to expose a new #+BEGIN_SRC option called :async. If the new :async option is specified in a block code for the Clojure language, then that code block will be processed asynchronously. What this means is that a new window will be created in Emacs, it will be populated with anything that is outputted to the REPL and then it will be closed once the processing will be finalized.

Here is an example of a code block that would use that new option:

#+BEGIN_SRC clojure :results output :async

(dotimes [n 10]
  (println n ".")
  (Thread/sleep 500))

#+END_SRC

This code would output “1. 2.” etc into a new window and would close that window when it reaches 10 and then populate the #+RESULTS section with the output of the code.

This code works with the :results options output, value and silent. If output is specified, then everything that was outputted into the window will be added into the results section of the code block. If value is specified, then all output will still be displayed into the window, but only the resulting value will be added to the results section of the code block. If silent is specified, then all the output will still be displayed into the window, but nothing will be displayed in the results section of the code block.

If the :async is omitted, then the normal behavior of Org-babel-clojure will be used, with the new timeout setting org-babel-clojure-nrepl-timeout.

As I mentioned above, this is my attempt in coding something for Emacs using Elisp. There are certainly things that should be done differently. So if you have any Elisp and/or Cider/nREPL knowledge, and if you have some time to review this code, I am sure we could improve the usage of this function. The only thing I know is that such asynchronous capabilities of the Clojure code blocks is essential.

There is one major area of improvement that I noted. Right now, the results comes asynchronously, but we still can’t use the Emacs instance to do other things (like writing in the Org-mode file while the process is running in background and results reported in this other buffer. Until this other issue is resolved, I don’t think we can say that this really makes Org-babel-clojure really 100% asynchronous. If this can be done (I did not have time to look into this yet), then I think the :async feature would be fully and properly integrated, but I am not yet sure if this is possible.