{"id":3227,"date":"2016-06-16T16:49:54","date_gmt":"2016-06-16T20:49:54","guid":{"rendered":"http:\/\/fgiasson.com\/blog\/?p=3227"},"modified":"2016-11-17T13:46:43","modified_gmt":"2016-11-17T18:46:43","slug":"improving-org-babel-clojure","status":"publish","type":"post","link":"https:\/\/fgiasson.com\/blog\/index.php\/2016\/06\/16\/improving-org-babel-clojure\/","title":{"rendered":"Improving org-babel-clojure"},"content":{"rendered":"

In a previous blog post, I started to play with org-babel-clojure<\/a> to improve its capabilities such that Clojure gets better integrated into Org-mode for creating notebooks and Literate programs. The first thing I wanted to do is to remove the 20 seconds timeout that was defaulted with the nrepl<\/code>. That meant that it was not possible to run procedures for longer than 20 seconds before it died with a timeout. Once this was implemented, the next step was to add a new feature to see the underlying process of a code block. Because the nature of my work (extensive work with big datasets), my procedures take time to run (minutes\u00e2\u20ac\u00a6 hours\u00e2\u20ac\u00a6) and much information [about the process] is output to the terminal. However, in the org-babel-clojure<\/code> implementation, you had to wait until the code was executed before being able to see the processing. What I did at the time is to add a new :async<\/code> code block parameter which told org-babel-clojure<\/code> to output all the output of the nrepl<\/code>, when it was being processed, in a new window.<\/p>\n

That worked like a charm. However, after much interaction with Nicolas Goaziou, one of the core maintainers of Org-mode, it was clear that my implementation was not an asynchronous implementation but really just a live processing output.<\/p>\n

At the same time, I did find another major irritant: if an exception was raised in my Clojure code, then nothing was output to Org-mode, it was simply silently dying. The only way to see the exception was to switch to the Clojure major mode (using C-c '<\/code>) and to rerun the code block.<\/p>\n

<\/p>\n

Here are the two new improvements to my org-babel-clojure<\/code> implementation:<\/p>\n

    \n
  1. Rename the :async<\/code> block parameter to :show-process<\/code><\/li>\n
  2. Output the exceptions and errors messages with the output<\/code> and the value<\/code> results parameter<\/li>\n<\/ol>\n

    By renaming to :show-process<\/code> I remove the ambiguity of the feature. Eventually we should get to a real asynchronous process, but the issue is that it is much more complex than I initially thought and this is a problem being addressed in Org-mode for all backends and not just org-babel-clojure<\/code>.<\/p>\n

    Then every exception or error messages returned by nrepl<\/code> are appended the value<\/code> or the output<\/code> returned by the code block. That way, we immediately see that something is going wrong directly within Org-mode.<\/p>\n

    Here is the latest version of my org-mode-babel implementation:<\/p>\n

    \n
    (<\/span>defvar<\/span> nrepl-sync-request-timeout<\/span>)<\/span>\n\n(<\/span>defun<\/span> org-babel-execute:clojure<\/span> (<\/span>body params)<\/span>\n  \"Execute a block of Clojure code with Babel. The block can be executed<\/span>\n   synchenously by default or asynchronously with the :show-process parameter\"<\/span>\n  (<\/span>let<\/span> (<\/span>(<\/span>expanded (<\/span>org-babel-expand-body:clojure body params)<\/span>)<\/span>\n        (<\/span>sbuffer \"*Clojure Show Process Sub Buffer*\"<\/span>)<\/span>\n        (<\/span>show (<\/span>if<\/span> (<\/span>assoc :show-process<\/span> params)<\/span> t nil)<\/span>)<\/span>\n        (<\/span>response (<\/span>cons 'dict nil)<\/span>)<\/span>\n        status\n        result)<\/span>\n    (<\/span>case org-babel-clojure-backend\n      (<\/span>cider\n       (<\/span>require<\/span> 'cider<\/span>)<\/span>\n       (<\/span>let<\/span> (<\/span>(<\/span>result-params (<\/span>cdr (<\/span>assoc :result-params<\/span> params)<\/span>)<\/span>)<\/span>)<\/span>\n         ; <\/span>Check if the user want to run code asynchronously<\/span>\n         (<\/span>when<\/span> show\n           ; <\/span>Create a new window with the show output buffer<\/span>\n           (<\/span>switch-to-buffer-other-window sbuffer)<\/span>\n\n           ; <\/span>Run the Clojure code asynchronously in nREPL<\/span>\n           (<\/span>nrepl-request:eval\n            expanded \n            (<\/span>lambda<\/span> (<\/span>resp)<\/span> \n              (<\/span>when<\/span> (<\/span>member \"out\"<\/span> resp)<\/span>\n                ; <\/span>Print the output of the nREPL in the asyn output buffer<\/span>\n                (<\/span>princ (<\/span>nrepl-dict-get resp \"out\"<\/span>)<\/span> (<\/span>get-buffer sbuffer)<\/span>)<\/span>)<\/span>\n              (<\/span>when<\/span> (<\/span>member \"ex\"<\/span> resp)<\/span>\n                ; <\/span>In case there is an exception, then add it to the output \n                ; buffer as well<\/span>\n                (<\/span>princ (<\/span>nrepl-dict-get resp \"ex\"<\/span>)<\/span> (<\/span>get-buffer sbuffer)<\/span>)<\/span>\n                (<\/span>princ (<\/span>nrepl-dict-get resp \"root-ex\"<\/span>)<\/span> (<\/span>get-buffer sbuffer)<\/span>)<\/span>)<\/span>\n              (<\/span>when<\/span> (<\/span>member \"err\"<\/span> resp)<\/span>\n                ; <\/span>In case there is an error, then add it to the output \n                ; buffer as well<\/span>\n                (<\/span>princ (<\/span>nrepl-dict-get resp \"err\"<\/span>)<\/span> (<\/span>get-buffer sbuffer)<\/span>)<\/span>)<\/span>\n              (<\/span>nrepl--merge response resp)<\/span>\n              ; <\/span>Update the status of the nREPL output session<\/span>\n              (<\/span>setq status (<\/span>nrepl-dict-get response \"status\"<\/span>)<\/span>)<\/span>)<\/span>\n            (<\/span>cider-current-connection)<\/span> \n            (<\/span>cider-current-session)<\/span>)<\/span>\n\n           ; <\/span>Wait until the nREPL code finished to be processed<\/span>\n           (<\/span>while<\/span> (<\/span>not (<\/span>member \"done\"<\/span> status)<\/span>)<\/span>\n             (<\/span>nrepl-dict-put response \"status\"<\/span> (<\/span>remove \"need-input\"<\/span> status)<\/span>)<\/span>\n             (<\/span>accept-process-output nil 0.01)<\/span>\n             (<\/span>redisplay)<\/span>)<\/span>\n\n           ; <\/span>Delete the show buffer & window when the processing is finalized<\/span>\n           (<\/span>let<\/span> (<\/span>(<\/span>wins (<\/span>get-buffer-window-list sbuffer nil t)<\/span>)<\/span>)<\/span>\n             (<\/span>dolist<\/span> (<\/span>win wins)<\/span>\n               (<\/span>delete-window win)<\/span>)<\/span>\n             (<\/span>kill-buffer sbuffer)<\/span>)<\/span>\n\n           ; <\/span>Put the output or the value in the result section of the code block<\/span>\n           (<\/span>setq result (<\/span>concat (<\/span>nrepl-dict-get response \n                                                (<\/span>if<\/span> (<\/span>or \n                                                      (<\/span>member \"output\"<\/span> result-params)<\/span>\n                                                      (<\/span>member \"pp\"<\/span> result-params)<\/span>)<\/span>\n                                                    \"out\"<\/span>\n                                                  \"value\"<\/span>)<\/span>)<\/span>\n                                (<\/span>nrepl-dict-get response \"ex\"<\/span>)<\/span>\n                                (<\/span>nrepl-dict-get response \"root-ex\"<\/span>)<\/span>\n                                (<\/span>nrepl-dict-get response \"err\"<\/span>)<\/span>)<\/span>)<\/span>)<\/span>\n         ; <\/span>Check if user want to run code synchronously<\/span>\n         (<\/span>when<\/span> (<\/span>not show)<\/span>\n           (<\/span>setq response (<\/span>let<\/span> (<\/span>(<\/span>nrepl-sync-request-timeout \n                                 org-babel-clojure-sync-nrepl-timeout)<\/span>)<\/span>\n                            (<\/span>nrepl-sync-request:eval\n                             expanded (<\/span>cider-current-connection)<\/span> \n                                      (<\/span>cider-current-session)<\/span>)<\/span>)<\/span>)<\/span>\n           (<\/span>setq result\n                 (<\/span>concat \n                  (<\/span>nrepl-dict-get response (<\/span>if<\/span> (<\/span>or (<\/span>member \"output\"<\/span> result-params)<\/span>\n                                                   (<\/span>member \"pp\"<\/span> result-params)<\/span>)<\/span>\n                                               \"out\"<\/span>\n                                             \"value\"<\/span>)<\/span>)<\/span>\n                  (<\/span>nrepl-dict-get response \"ex\"<\/span>)<\/span>\n                  (<\/span>nrepl-dict-get response \"root-ex\"<\/span>)<\/span>\n                  (<\/span>nrepl-dict-get response \"err\"<\/span>)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>\n       (<\/span>slime\n        (<\/span>require<\/span> 'slime<\/span>)<\/span>\n        (<\/span>with-temp-buffer<\/span>\n          (<\/span>insert expanded)<\/span>\n          (<\/span>setq result\n                (<\/span>slime-eval\n                 `(<\/span>swank:eval-and-grab-output\n                   ,(<\/span>buffer-substring-no-properties (<\/span>point-min)<\/span> (<\/span>point-max)<\/span>)<\/span>)<\/span>\n                 (<\/span>cdr (<\/span>assoc :package<\/span> params)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>\n      (<\/span>org-babel-result-cond (<\/span>cdr (<\/span>assoc :result-params<\/span> params)<\/span>)<\/span>\n        result\n        (<\/span>condition-case<\/span> nil (<\/span>org-babel-script-escape result)<\/span>\n          (<\/span>error<\/span> result)<\/span>)<\/span>)<\/span>)<\/span>)<\/span>\n<\/pre>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
    In a previous blog post, I started to play with org-babel-clojure to improve its capabilities such that Clojure gets better integrated into Org-mode for creating notebooks and Literate programs. The first thing I wanted to do is to remove the 20 seconds timeout that was defaulted with the nrepl. That meant that it was not […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[251,279,277,66],"tags":[278,252,254,276],"class_list":["post-3227","post","type-post","status-publish","format-standard","hentry","category-clojure","category-emacs","category-literate-programming","category-programming","tag-cider","tag-clojure-2","tag-emacs","tag-orgmode"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3227","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=3227"}],"version-history":[{"count":6,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3227\/revisions"}],"predecessor-version":[{"id":3506,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/posts\/3227\/revisions\/3506"}],"wp:attachment":[{"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=3227"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=3227"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fgiasson.com\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=3227"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}