Tag Archive for 'emacs'

Improving org-babel-clojure

In a previous blog post, I started to play with org-babel-clojure to improve its capabilities such that Clojure gets better integrated into Org-mode for creating notebooks and Literate programs. The first thing I wanted to do is to remove the 20 seconds timeout that was defaulted with the nrepl. That meant that it was not possible to run procedures for longer than 20 seconds before it died with a timeout. Once this was implemented, the next step was to add a new feature to see the underlying process of a code block. Because the nature of my work (extensive work with big datasets), my procedures take time to run (minutes… hours…) and much information [about the process] is output to the terminal. However, in the org-babel-clojure implementation, you had to wait until the code was executed before being able to see the processing. What I did at the time is to add a new :async code block parameter which told org-babel-clojure to output all the output of the nrepl, when it was being processed, in a new window.

That worked like a charm. However, after much interaction with Nicolas Goaziou, one of the core maintainers of Org-mode, it was clear that my implementation was not an asynchronous implementation but really just a live processing output.

At the same time, I did find another major irritant: if an exception was raised in my Clojure code, then nothing was output to Org-mode, it was simply silently dying. The only way to see the exception was to switch to the Clojure major mode (using C-c ') and to rerun the code block.

Here are the two new improvements to my org-babel-clojure implementation:

  1. Rename the :async block parameter to :show-process
  2. Output the exceptions and errors messages with the output and the value results parameter

By renaming to :show-process I remove the ambiguity of the feature. Eventually we should get to a real asynchronous process, but the issue is that it is much more complex than I initially thought and this is a problem being addressed in Org-mode for all backends and not just org-babel-clojure.

Then every exception or error messages returned by nrepl are appended the value or the output returned by the code block. That way, we immediately see that something is going wrong directly within Org-mode.

Here is the latest version of my org-mode-babel implementation:

(defvar nrepl-sync-request-timeout)

(defun org-babel-execute:clojure (body params)
  "Execute a block of Clojure code with Babel. The block can be executed
   synchenously by default or asynchronously with the :show-process parameter"
  (let ((expanded (org-babel-expand-body:clojure body params))
        (sbuffer "*Clojure Show Process Sub Buffer*")
        (show (if (assoc :show-process params) t nil))
        (response (cons 'dict nil))
        status
        result)
    (case org-babel-clojure-backend
      (cider
       (require 'cider)
       (let ((result-params (cdr (assoc :result-params params))))
         ; Check if the user want to run code asynchronously
         (when show
           ; Create a new window with the show output buffer
           (switch-to-buffer-other-window sbuffer)

           ; Run the Clojure code asynchronously in nREPL
           (nrepl-request:eval
            expanded 
            (lambda (resp) 
              (when (member "out" resp)
                ; Print the output of the nREPL in the asyn output buffer
                (princ (nrepl-dict-get resp "out") (get-buffer sbuffer)))
              (when (member "ex" resp)
                ; In case there is an exception, then add it to the output 
                ; buffer as well
                (princ (nrepl-dict-get resp "ex") (get-buffer sbuffer))
                (princ (nrepl-dict-get resp "root-ex") (get-buffer sbuffer)))
              (when (member "err" resp)
                ; In case there is an error, then add it to the output 
                ; buffer as well
                (princ (nrepl-dict-get resp "err") (get-buffer sbuffer)))
              (nrepl--merge response resp)
              ; Update the status of the nREPL output session
              (setq status (nrepl-dict-get response "status")))
            (cider-current-connection) 
            (cider-current-session))

           ; Wait until the nREPL code finished to be processed
           (while (not (member "done" status))
             (nrepl-dict-put response "status" (remove "need-input" status))
             (accept-process-output nil 0.01)
             (redisplay))

           ; Delete the show buffer & window when the processing is finalized
           (let ((wins (get-buffer-window-list sbuffer nil t)))
             (dolist (win wins)
               (delete-window win))
             (kill-buffer sbuffer))

           ; Put the output or the value in the result section of the code block
           (setq result (concat (nrepl-dict-get response 
                                                (if (or 
                                                      (member "output" result-params)
                                                      (member "pp" result-params))
                                                    "out"
                                                  "value"))
                                (nrepl-dict-get response "ex")
                                (nrepl-dict-get response "root-ex")
                                (nrepl-dict-get response "err"))))
         ; Check if user want to run code synchronously
         (when (not show)
           (setq response (let ((nrepl-sync-request-timeout 
                                 org-babel-clojure-sync-nrepl-timeout))
                            (nrepl-sync-request:eval
                             expanded (cider-current-connection) 
                                      (cider-current-session))))
           (setq result
                 (concat 
                  (nrepl-dict-get response (if (or (member "output" result-params)
                                                   (member "pp" result-params))
                                               "out"
                                             "value"))
                  (nrepl-dict-get response "ex")
                  (nrepl-dict-get response "root-ex")
                  (nrepl-dict-get response "err"))))))
       (slime
        (require 'slime)
        (with-temp-buffer
          (insert expanded)
          (setq result
                (slime-eval
                 `(swank:eval-and-grab-output
                   ,(buffer-substring-no-properties (point-min) (point-max)))
                 (cdr (assoc :package params)))))))
      (org-babel-result-cond (cdr (assoc :result-params params))
        result
        (condition-case nil (org-babel-script-escape result)
          (error result)))))

My Optimal GNU Emacs Settings for Developing Clojure (Revised)

It has been 2 years since I last revised my optimal GNU Emacs settings for developing Clojure. Since then, many things have changed in the Cider/Emacs ecosystem and it is why a revision of these settings is now warranted.

There is one set of settings that I will not discuss in this blog post, and these are all the settings related to Literate Programming using Org-mode. Since much needs to be said about these, they will be the topic of a subsequent blog post that will focus exclusively on that matter.

GNU Emacs

GNU Emacs is an awesome programming interface. It is said to be the programmable programming interface. It is terrific but it comes at a price: steep learning curve and much time spent testing and configuring useful packages. However, once these hurdles are passed, the sun starts to shine and a joyful journey begins.

I am personally developing on Windows 10, but these steps should be platform agnostic. You only have to download and install the latest GNU Emacs 24 version.

The first thing you have to do is to locate your .emacs file. All the configurations I am defining in this blog post goes into that file.

Packages

Once Emacs is installed, the first thing you have to do is to install all the packages that are required to develop in Clojure or that will make your life easier for handling the code. The packages that you have to install are:

  1. cider
    • Clojure Integrated Development Environment and REPL. This is the actual Clojure IDE. It includes everything to work with Clojure in Emacs. It runs the REPL, it has a debugger, etc.
  2. company
    • In-buffer auto-completion framework. This is now the best integrated auto-complete for Cider.
  3. monokai-theme
    • This is a new theme that is pleasant to look at and that works pretty well with Clojure code.
  4. rainbow-delimiters
    • Highlight nested parenthesis, brackets, braces a different color at each depth – This is really handy to visually see where you are with your parenthesis. An essential to have (in my view).
  5. highlight-symbol
    • Highlight all occurrences of a symbol in a buffer. This is really handy to find occurrences of variables, or function calls, etc.
  6. heml-ag
    • High performance file system search function. This is essential to find all occurrences of a [regex] pattern within local files.

Before installing them, we have to tell Emacs which package repositories it has access to so as to find these packages. At the top of your .emacs file, put:

;; Define packages archives repositories
(require 'package)

(add-to-list 'package-archives
  '("melpa-stable" . "http://melpa-stable.milkbox.net/packages/"))

(add-to-list 'package-archives 
  '("marmalade" . "https://marmalade-repo.org/packages/"))  

;; Initialize all the ELPA packages (what is installed using the packages commands)    
(package-initialize)

If you are editing your .emacs file directly into Emacs, and you can re-evaluate the settings file using Emacs, then by moving the cursor at each top-level expression end (after closing parenthesis) and press C-x C-e. However, it may be faster just to close and restart Emacs to take the new settings into account. You can use any of these methods for the following set of settings changes.

Before changing any more settings, we will first install all the required packages using the following sequence of commands:

M-x package-install [RET] cider [RET]
M-x package-install [RET] company [RET]
M-x package-install [RET] monokai-theme [RET]
M-x package-install [RET] rainbow-delimiters]] [RET]
M-x package-install [RET] highlight-symbol [RET]
M-x package-install [RET] helm-ag [RET]

Additionally, you could have used M-x package-list-packages, then move your cursor in the buffer to the packages’ line. Then press i (for install) and once all the packages are selected, you could have press x (execute) to install all the packages all at once.

In the list of commands above, M-x is the “meta-key” normally bound to the left Alt key on your keyboard. So, M-x usually means Alt-x.

Now that all the packages are installed, let’s take a look at how we should configure them.

Installing Leiningen or Boot

An important piece of software to install is a build tool for Clojure. The two most popular choices are Leiningen and Boot.

Installing Platinum Searcher

For the helm-ag package, you will have to install the Platinum Searcher application. Installing it on Windows is quite simple. Just download the proper package, and extract the pt.ex file somewhere on your computer (let’s say in c:/platinum-searcher). We will configure it later.

Configuring Keyboard

If you are using an English/US keyboard, you can skip this section. Since I use a French Canadian layout (On an English/US Das Keyboard!), I had multiple issues to have my keys working since all the binding changed in Emacs. To solve this problem, I simply had to define that language configuration option. Then I had to start using the right Alt key of my keyboard to write my brackets, curly brackets, etc:

;; Enable a Canadian French keyboard layout
(require 'iso-transl)

Configure Text Editor

There are a few settings we have to specify related to the Emacs text editor. We want to send the comment columns of Clojure comments to column 70 and we want to force the usage of UTF-8 as the default encoding. Finally we want to start emacs maximized in Windows:

;; UTF-8 as default encoding
(set-language-environment "UTF-8")

;; Set the default comment column to 70
(setq-default comment-column 70)

;; Every time a window is started, make sure it get maximized
(add-to-list 'default-frame-alist '(fullscreen . maximized))

Configuring Fonts

There is a new wonderful coding font that I discovered recently called the Input (Font for Code). This is a really neat font that works particularly well. You just have to go to their site, define the characteristics you want for it, download and install it locally.

Once it is installed on your system, configure it that way in .emacs:

;; Use the Input font size 12
(set-default-font "Input-12")

Cider

Now that the general Emacs settings are configured, let’s focus on Cider. We want to make sure that we enter in cider mode when entering in Clojure major mode. Then we have to specify that we want to use company-mode as the auto-completion framework and finally we want to specify that when we enter via the client in Cider, that we want a new line and to indent that new line according to where we are in the code.

;; Enter cider mode when entering the clojure major mode
(add-hook 'clojure-mode-hook 'cider-mode)

;; Turn on auto-completion with Company-Mode
(global-company-mode)
(add-hook 'cider-repl-mode-hook #'company-mode)
(add-hook 'cider-mode-hook #'company-mode)

;; Replace return key with newline-and-indent when in cider mode.
(add-hook 'cider-mode-hook '(lambda () (local-set-key (kbd "RET") 'newline-and-indent)))

Show Parenthesis Mode

Another handy feature is to enable, by default, the show-parent-mode configuration option. That way, every time the cursor points to a parenthesis, the parent parenthesis will be highlighted into the user interface. This is an essential must-have with Par Edit:

;; Show parenthesis mode
(show-paren-mode 1)

Rainbow Delimiters

Another essential package to have helps you out in maintaining these parenthesis. The rainbow delimiters will change the color of the parenthesis depending on how “deep” they are into the structure. Another essential visual cue:

;; rainbow delimiters
(add-hook 'prog-mode-hook #'rainbow-delimiters-mode)

Monokai Theme

The Monokai theme is joyful and really fun to work Clojure code with. This is the current theme I use.

;; Set theme & font size
(add-to-list 'custom-theme-load-path "~/.emacs.d/lib/monokai-theme")
(load-theme 'monokai t)

There is another configuration we need to do which is to make sure that the theme’s colors are used by company-mode such that the popup contextual menus are properly themed:

;;
;; Update the color of the company-mode context menu to fit the Monokai theme
;; @source: https://github.com/search?q=deftheme+company-tooltip&type=Code
;;
(deftheme monokai-overrides)

(let ((class '((class color) (min-colors 257)))
      (terminal-class '((class color) (min-colors 89))))

  (custom-theme-set-faces
   'monokai-overrides

   ;; Linum and mode-line improvements (only in sRGB).
   `(linum
     ((,class :foreground "#75715E"
              :background "#49483E")))
   `(mode-line-inactive
     ((,class (:box (:line-width 1 :color "#2c2d26" :style nil)
                    :background "#2c2d26"))))

   ;; Custom region colouring.
   `(region
     ((,class :foreground "#75715E"
              :background "#49483E")
      (,terminal-class :foreground "#1B1E1C"
                       :background "#8B8878")))

   ;; Additional modes
   ;; Company tweaks.
   `(company-tooltip-common
     ((t :foreground "#F8F8F0"
         :background "#474747"
         :underline t)))

   `(company-template-field
     ((t :inherit company-tooltip
         :foreground "#C2A1FF")))

   `(company-tooltip-selection
     ((t :background "#349B8D"
         :foreground "#BBF7EF")))

   `(company-tooltip-common-selection
     ((t :foreground "#F8F8F0"
         :background "#474747"
         :underline t)))

   `(company-scrollbar-fg
     ((t :background "#BBF7EF")))

   `(company-tooltip-annotation
     ((t :inherit company-tooltip
         :foreground "#C2A1FF")))

   ;; Popup menu tweaks.
   `(popup-menu-face
     ((t :foreground "#A1EFE4"
         :background "#49483E")))

   `(popup-menu-selection-face
     ((t :background "#349B8D"
         :foreground "#BBF7EF")))

   ;; Circe
   `(circe-prompt-face
     ((t (:foreground "#C2A1FF" :weight bold))))

   `(circe-server-face
     ((t (:foreground "#75715E"))))

   `(circe-highlight-nick-face
     ((t (:foreground "#AE81FF" :weight bold))))

   `(circe-my-message-face
     ((t (:foreground "#E6DB74"))))

   `(circe-originator-face
     ((t (:weight bold))))))

helm-ag

It is essential to have a local file system search functionality. This is what helm-ag provides. I configured it to use the Platinum Searcher. I also bound it to the key M-s for easy invocation:

;; Configure helm-ag
;; Make sure to have Platinum Searcher installed: https://github.com/monochromegane/the_platinum_searcher

(custom-set-variables
 '(helm-ag-base-command "C:/platinum-searcher/pt -e --nocolor --nogroup"))

(global-set-key (kbd "M-s") 'helm-do-ag)

Syntax Highlighting

It is often handy to be able to highlight symbols within a buffer. I use highlight-symbol for that purpose but I re-mapped its functionalities to keys more meaningful and natural to me:

; Syntax Highlighting
(require 'highlight-symbol)
(global-set-key (kbd "C-é") 'highlight-symbol-at-point)
(global-set-key (kbd "C-.") 'highlight-symbol-next)
(global-set-key (kbd "C-,") 'highlight-symbol-prev)
(global-set-key (kbd "C-;") 'highlight-symbol-query-replace)

Binding Some Keys

I like to bind some behaviors to the F-keys. What I want is to be able to run Cider and to switch frames (windows within monitors) in a single click. I also added a shortkey for starting speedbar for the current buffer; it is an essential for managing project files. What I did is to bind these behaviors to these keys:

(global-set-key [f9] 'cider-jack-in)
(global-set-key [apps] 'other-frame)
(global-set-key [f11] 'speedbar)

Fixing the Scroll

There is one thing that I really didn’t like, and it was the default behavior of the scrolling of Emacs on Windows. After some searching, I found the following configurations that I could fix to have a smoother scrolling behavior on Windows:

;; scroll one line at a time (less "jumpy" than defaults)
(setq mouse-wheel-scroll-amount '(1 ((shift) . 1))) ;; one line at a time
(setq mouse-wheel-progressive-speed nil) ;; don't accelerate scrolling
(setq mouse-wheel-follow-mouse 't) ;; scroll window under mouse
(setq scroll-step 1) ;; keyboard scroll one line at a time

Conclusion

The Emacs/Clojure ecosystem has improved quite a lot in the last two years thanks to Cider and the decision they made in its development. The configuration of Cider is now much simpler with company-mode. This blog post provides the latest improvement I did to my setup.

As I said in my introduction, there is a whole bunch of settings that I did not include in this blog post related to configuring org-mode for doing Literate Programming in Clojure. I will discuss these in its own blog post, likely to be published within the next week.

Literate Programming and Team Development

I recently wrote a post about how I was using Literate Programming principles in Org-mode to write the unit tests of my applications side-by-side with the code it tests. I got some good feedbacks about the post, however one that particularly caught my eye is a blog post on the Irreal blog which states:

One possible problem with this procedure is if you’re working in a team and not everyone is an Emacs user. Non Emacs users won’t be keen to tangle an Org document to get the code and probably won’t feel comfortable making changes to an Org file. Unless everyone you’re working with is an Emacs user or you’re working alone, this means that Giasson’s workflow will have to be limited to initial development. Still, it’s a powerful technique and well worth experimenting with.

It is not the first time I have read such observation about Literate Programming (and Org-mode & Emacs). This is certainly a right and legitimate concern. Literate Programming is a set of concepts and principles with two actions at its core: weaving (generating human readable version of the application) and tangling (generating the computer code of the application) [a literate document]. The literate document has to comply with some syntax to determine what needs to be weaved and what needs to be tangled. This syntax needs to be implemented in a development environment to be usable. It could be fully integrate (like with Org-mode) or part of a development workflow (like CWEB). It is for this reason that there may be a possible problem: because the way to write a software application is inherently different than how we learned to program and it requires some specific development environments or workflows. This is fine if there is only one developer on the application or only a few that already use the same environment.

The real concern arises when Literate Programming with a specific Development Environment is forced upon an existing [potentially large] team of developers. The nature of humans, the diversity of backgrounds, knowledge and personalities would probably doom such a project. The willingness won’t be the same, some won’t want to learn a new way to program, etc.

However not all situations are the same. If a new team needs to be hired, then the task is completely different. The skills required to develop using the Literate Programming paradigm, or more importantly their willingness to learn these new skills, could be central to the selection process.

Emerging architectures like Micro-services may also help adopting Literate Programming in existing projects. Micro-services architecture enables all kind of web services, developed in all kind of languages, to interact together like if it was a monolithic application. In such a setup, the code of already developed micro-services could remain the same and then new micro-service initiatives could be developed using Literate Programming by a dedicated person, or a small dedicated team without affecting any other development teams. The usage of these new development techniques would be constrained to these new micro-services projects.

However I am conscious that Literate Programming is a new development paradigm that needs much effort, concentration and willingness to adopt. It changes the core development process of a developer which can be quite destabilizing and daunting. It is a shift in development culture that is required here. This is not a simple and easy thing. This is why I agree that team development using Literate Programming is a legitimate concern to have.

However, I am finding that the benefits of this cultural shift outweigh the initial investment that is required (and which is non-trivial). That is why I will continue to document the way I develop my new applications in Literate Programming using Org-mode and to explain all of the benefits I am seeing and the issues I may face.

In mean time, if you are interested in Literate Programming, I would strongly suggest you to take 30 minutes to listen at Thimoty Daly‘s Literate Programming in the Large talk:

Another work I want to highlight is Karsten Schmidt‘s work at thi-ng. All of his projects are using Org-mode. His biggest project is geom which has hundred of stars, 22 forks and 6 contributors which suggest that Org-mode is not a show stopper to create a healthy opensource project.




This blog is a regularly updated collection of my thoughts, tips, tricks and ideas about data mining, data integration, data publishing, the semantic Web, my researches and other related software development.


RSS Twitter LinkedIN


Follow

Get every new post on this blog delivered to your Inbox.

Join 92 other followers:

Or subscribe to the RSS feed by clicking on the counter:




RSS Twitter LinkedIN