RDF Code: Serializing RDF Data as Clojure Code

RDF Code is a specification to serialize RDF data as Clojure code1. This blog post introduce the first version of this new RDF serialization format. I will outline the rules that specify how such RDF data should be serialized using the Clojure programming language.

This specification may change over time. However, this is the specification that will be used for the future blog posts that I will write about this subject, and for the code that will be released.

I am also expecting feedbacks and propositions to make this serialization easier to use, simpler to define and cleaner.

What we do with this serialization is to write RDF resources as Clojure maps. This is not about defining a DSL to manipulate this data, but really to define the core Clojure structure that will be manipulated by Clojure functions and applications. Eventually a (or multiple) DSL could be created to help users and developers using this RDF data in their Clojure application. But this is not the current focus.

A Complete RDF Resource

Before outlining all the rules to create well-formed RDF data as Clojure code, let’s take a look at a resource that uses all the serialization features2. Note that this is the more complex it can be. Below we will see how we can normalize the usage of the serialization rules to end up with clean and easy to read RDF data as Clojure code.

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type [foaf/+person owl/+thing]
iron/pref-label “Fred”
iron/alt-label {value “Frederick”
lang “en”}
foaf/skypeID {value “frederick.giasson”
datatype xsd/*string}
foaf/knows [{uri “http://foo.com/datasets/people/bob”}
mike
“http://foo.com/datasets/people/teo”]})[/raw]
[/cc]

This code shows how to:

  • Serialize a single resource as a Clojure map
  • How to define a URI for that resource
  • How to define one or multiple rdf:type for a resource
  • How to define one or multiple values for a owl:DatatypeProperty
  • How to define one or multiple values for a owl:ObjectProperty
  • How to define the language of a Literal
  • How to define the datatype of a Literal

As you can see, such a RDF serialization format is expressive enough to be able to express any RDF triples. It also has syntactic rules that help reading and writing RDF data in that format.

Note that since this RDF data is also Clojure code, it means that the serialization format has been highly influenced by Clojure’s own syntax and coding style3. RDF data serialized using this format also needs to be valid Clojure code. Now, let’s outline the rules that govern the creation of such RDF data, then let explains all these rules using simple RDF code examples.

RDF Code Rules

Here is the list of all the rules that govern the creation of RDF data serialized as Clojure code:

  1. A RDF resource is defined as a Clojure map where:
    1. Every key is a symbol that reference a function
    2. Every value is a:
      1. string
        1. A string is considered a literal if the key is a owl:DatatypeProperty
        2. A string is considered a URI if the key is a owl:ObjectProperty
      2. map
        1. A map represent a literal if the value key is present
        2. A map represent a reference to another resource if the uri key is present
        3. A map is invalid if it doesn’t have a uri nor a value key
      3. vector
        1. A vector refer to multiple values. Values of a vector can be strings, maps or symbols
      4. symbol
        1. A symbol can be created to simplify the serialization. However, these symbols have to reference a string or a map object

In addition to these rules, there are some more specific rules such as:

  1. The value of a uri key is always a string
  2. If the rdf/type key is not defined for a resource, then the resource is considered to be of type owl:Thing (since everything is at least an instance of the owl:Thing class in OWL)

Finally, there are two additional classes and datatypes creation conventions:

  1. The name of the classes starts with a + sign, like: owl/+thing
  2. The name of the datatypes starts with a * sign, like: xsd/*string

As you can see, the rules that govern the serialization of RDF data as Clojure code are minimal and should be simple to understand for someone who is used to Clojure code and that tried to write a few resource examples using this format. Now, let’s apply these rules with a series of examples.

Note 1: in the examples of this blog post, I am referring to symbols like uri, value, lang, datatype, etc. To make the rules simpler to read and understand, consider that these symbols are defined in the user‘s namespace. However, they are symbols that are defined in the rdf.core namespace that will be made publicly available later.

Note 2: I am also referring to namespaced symbols like rdf/type, iron/pref-label, etc. These symbols are defined in their respective namespaces. They have been defined such as (:require [ontologies.rdf :as rdf] [ontologies.iron :as iron]) into the ns function of the Clojure source code file that define this RDF resource. We will discuss about these namespaces in a subsequent blog post.

Serializing RDF Code in N-Triples

Before starting to list all the examples, let’s define a Clojure function that we will use to convert the RDF code as N-Triples4. N-Triples is just a list of <subject> <predicate> <object> triples that describes the RDF resources we are describing. N-Triples is the simplest and most verbose RDF serialization that currently exists. What this serialize-ntriples function does is to take a Clojure map that represent a RDF resource and return a string that represents the serialized N-Triples.

What is important with that code is that is shows how the rules we outlined above got implemented to serialize RDF code as N-Triples. Such serializer function could be created to serialize Turtle and XML RDF serializations as well. Note that you won’t be able to use the serialize-ntriples function because you are missing the ontologies files. I will make them available in a subsequent blog post that will explain how properties, classes and datatypes are created and used in this context.

[cc lang=’lisp’ line_numbers=’false’]
[raw](declare serialize-ntriples-map-value serialize-ntriples-string-value is-datatype-property?)

(defn serialize-ntriples
[resource]
(let [n3 (atom “”)
iri (get resource rdf.core/uri)]
(doseq [[property vals] resource]
(let [property-uri (get (meta property) rdf.core/uri)]
; Don’t do anything with the “uri” key
(if (not= property rdf.core/uri)
(if (vector? vals)
; Here the value is a vector of maps or values
(doseq [val vals]
(if (map? val)
; The value of the vector is a map
(reset! n3 (str @n3 (serialize-ntriples-map-value val iri property-uri)))
(if (string? val)
; The value of the vector is a string
(reset! n3 (str @n3 (serialize-ntriples-string-value val iri property-uri property))))))
(if (map? vals)
; The value of the property is a map
(reset! n3 (str @n3 (serialize-ntriples-map-value vals iri property-uri)))
(if (string? vals)
; The value of the property is some kind of literal
(reset! n3 (str @n3 (serialize-ntriples-string-value vals iri property-uri property)))))))))
@n3))

(defn- serialize-ntriples-map-value
[m iri property-uri]
(if (not (nil? (get m rdf.core/uri)))
; The value is a reference to another resource
(format “<%s> <%s> <%s> .\n” iri property-uri (get m rdf.core/uri))
(if (not (nil? (get m rdf.core/value)))
; The value is some kind of literal
(let [value (get m rdf.core/value)
lang (if (get m rdf.core/lang) (str “@” (get m rdf.core/lang)) “”)
datatype (if (get m rdf.core/datatype) (str “^^<” (get (get m rdf.core/datatype) rdf.core/uri) “>”) “”)]
(format “<%s> <%s> \”\”\”%s\”\”\”%s%s .\n” iri property-uri value lang datatype))
(if (string? m)
; The value of the sector is some kind of literal
(format “<%s> <%s> \”\”\”%s\”\”\” .\n” iri property-uri m)))))

(defn- serialize-ntriples-string-value
[s iri property-uri property]
; The value of the vector is a string
(if (true? (is-datatype-property? property))
; The property referring to this value is a owl:DatatypeProperty
(format “<%s> <%s> \”\”\”%s\”\”\” .\n” iri property-uri s)
; The property referring to this value is a owl:ObjectProperty
(format “<%s> <%s> <%s> .\n” iri property-uri s)))

(defn is-datatype-property?
[property]
(if (= (get (get (meta property) rdf/type) rdf.core/uri) (get ontologies.owl/+datatype-property rdf.core/uri))
(eval true)
(eval false)))[/raw]
[/cc]

The Simplest Resource

Here is the simplest resource that can be written:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type “http://xmlns.com/foaf/0.1/Person”
iron/pref-label “Fred”
foaf/skypeID “frederick.giasson”
foaf/knows [“http://foo.com/datasets/people/bob”
“http://foo.com/datasets/people/mike”
“http://foo.com/datasets/people/teo”]})[/raw]
[/cc]

It produces these triples:

[raw]

<http://foo.com/datasets/people/fred> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person> .
<http://foo.com/datasets/people/fred> <http://purl.org/ontology/iron#prefLabel> """Fred""" .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/skypeID> """frederick.giasson""" .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/bob> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/mike> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/teo> .

[/raw]

In this example, all the references to the classes and to other resources are made using strings that represent URIs. Then all the literal values are normal strings as well. The vector is composed of a list of URIs without mixing different type of values.

Using Classes Symbols

In this example, we will use the symbol that reference a class resource we have defined in an ontology:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type foaf/+person
iron/pref-label “Fred”
foaf/skypeID “frederick.giasson”
foaf/knows [“http://foo.com/datasets/people/bob”
“http://foo.com/datasets/people/mike”
“http://foo.com/datasets/people/teo”]})[/raw]
[/cc]

The same set of triples as the previous example will be generated by serialize-ntriples.

What is interesting with this new example is that it is more appealing to human readers. Instead of having a full URI string, we are seeing a symbol which is more pleasant to the eyes.

But the real benefit is no that it is more pleasant to the eyes, but that it really refers to something. It refers to a class resource. This means that we have a docstring for that class, that we can check the code that describes the class to see all its characteristics, that you will be able to auto-complete it in your IDE, etc.

Multiple Types

It is possible to define multiple types for a resource. The only thing you have to do is to use a vector as the value of the rdf/type key:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type [foaf/+person owl/+thing]
iron/pref-label “Fred”
foaf/skypeID “frederick.giasson”
foaf/knows [“http://foo.com/datasets/people/bob”
“http://foo.com/datasets/people/mike”
“http://foo.com/datasets/people/teo”]})[/raw]
[/cc]

This code will generate the following triples:

[raw]

<http://foo.com/datasets/people/fred> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person> .
<http://foo.com/datasets/people/fred> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2002/07/owl#Thing> .
<http://foo.com/datasets/people/fred> <http://purl.org/ontology/iron#prefLabel> """Fred""" .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/skypeID> """frederick.giasson""" .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/bob> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/mike> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/teo> .

[/raw]

Specifying a Datatype for a Literal

It is also possible to define a datatype for a literal. What you have to do is to use a map with a value and a datatype key:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type [foaf/+person owl/+thing]
iron/pref-label {value “Fred”
datatype xsd/*string}
foaf/skypeID “frederick.giasson”
foaf/knows [“http://foo.com/datasets/people/bob”
“http://foo.com/datasets/people/mike”
“http://foo.com/datasets/people/teo”]})[/raw]
[/cc]

The triples that will be generated are:

[raw]

<http://foo.com/datasets/people/fred> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person> .
<http://foo.com/datasets/people/fred> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2002/07/owl#Thing> .
<http://foo.com/datasets/people/fred> <http://purl.org/ontology/iron#prefLabel> """Fred"""^^<http://www.w3.org/2001/XMLSchema#string> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/skypeID> """frederick.giasson""" .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/bob> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/mike> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/teo> .

[/raw]

Specifying a Language Tag for a Literal

It is also possible to define a language tag for a literal. What you have to do is to use a map with a value and a lang key:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type [foaf/+person owl/+thing]
iron/pref-label {value “Fred”
lang “en”}
foaf/skypeID “frederick.giasson”
foaf/knows [“http://foo.com/datasets/people/bob”
“http://foo.com/datasets/people/mike”
“http://foo.com/datasets/people/teo”]})[/raw]
[/cc]

This code will produce the following triples:

[raw]

<http://foo.com/datasets/people/fred> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person> .
<http://foo.com/datasets/people/fred> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2002/07/owl#Thing> .
<http://foo.com/datasets/people/fred> <http://purl.org/ontology/iron#prefLabel> """Fred"""@en .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/skypeID> """frederick.giasson""" .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/bob> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/mike> .
<http://foo.com/datasets/people/fred> <http://xmlns.com/foaf/0.1/knows> <http://foo.com/datasets/people/teo> .

[/raw]

Referring to a URI Using a Map

It is possible to explicit the fact that a literal value is a URI reference. It is possible to do that by using a map and by specifying the uri key. Some people may prefer that approach because it makes the fact that a literal is a URI explicit without having to know the nature of the key (i.e. if the property is a datatype or an object property):

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type [foaf/+person owl/+thing]
iron/pref-label {value “Fred”
lang “en”}
foaf/skypeID “frederick.giasson”
foaf/knows [{uri “http://foo.com/datasets/people/bob”}
{uri “http://foo.com/datasets/people/mike”}
{uri “http://foo.com/datasets/people/teo”}]})[/raw]
[/cc]

The same triples will be generated as the example above.

Mixing Values in a Vector

It is possible to mix the values in a vector. In the following example, we are using literals as URIs, and maps that refer to URIs as well. The rules permit this kind of values mixing within a vector and the software that manipulate this kind of RDF data should be agnostic to this:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred {uri “http://foo.com/datasets/people/fred”
rdf/type [foaf/+person owl/+thing]
iron/pref-label {value “Fred”
lang “en”}
foaf/skypeID “frederick.giasson”
foaf/knows [“http://foo.com/datasets/people/bob”
{uri “http://foo.com/datasets/people/mike”}
“http://foo.com/datasets/people/teo”]})[/raw]
[/cc]

The same triples will be generated as the example above.

Using Symbols to Get Cleaner Code

Because of the way Clojure works, you can define new symbols that will refer to the values to add to this structure. Let’s go wild, and do define a symbol for each value of the resource we are describing:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def fred-uri “http://foo.com/datasets/people/fred”)

(def foaf_person foaf/+person)

(def owl_thing owl/+thing)

(def fred-label {value “Fred”
lang “en”})

(def fred-skype-id “frederick.giasson”)

(def bob-uri “http://foo.com/datasets/people/bob”)

(def mike-uri {uri “http://foo.com/datasets/people/mike”})

(def teo-uri “http://foo.com/datasets/people/teo”)

(def fred {uri fred-uri
rdf/type [foaf/+person owl/+thing]
iron/pref-label fred-label
foaf/skypeID fred-skype-id
foaf/knows [bob-uri
mike-uri
teo-uri]})[/raw]
[/cc]

The triples generated will be the same as the example above.

What happens here is that the Clojure reader substitutes the symbols by the actual things they refer to at evaluation time.

This is why this works and it is why it is still valid RDF code according to the rules outlined above. The serialize-ntriples function is not even aware that this data structure has been defined that way. It is the case because the map that it will receive as input will already be evaluated. This means that what it gets as input are the objects referenced by these symbols, and not the symbols themselves.

This mechanism can be leveraged to make the RDF code even more readable if you have patterns that a repeated constantly in a dataset file you are creating.

Conclusion

This is all good, but why yet another serialization for RDF? As I started to outline in my two previous blogposts on this topic, this is not just about serializing RDF in another format. It is about having RDF data that can be evaluated as code (in this case, Clojure code).

Considering RDF properties as functions open-up a World of possibilities. First, it means that by evaluating and compiling this kind of RDF code, you have a data structure that is able to validate itself according to the way the properties are defined into the OWL ontologies used to define their behaviors and semantics.

This blog post focused on how to serialize instances data. In the next blog posts, I will cover:

  • How to reify triples using this serialization
  • How to create the classes, properties and datatypes used by the serialization
  • How to validate the data structure
  • How to manage ontologies as Clojure packages

Feel free to comment this blog post and propose changes to the serialization format or the serialize-ntriples function.

Investigating Options to Serialize RDF data as Clojure Code

My initial intuition is that I could serialize RDF data into Clojure code where the OWL semantic of the RDF data is embedded, in some ways, into that code. I want to test how the general saying of homoiconic languages: Data as Code. Code as Data, fits with RDF & OWL.

Another intuition I have is the concept of Portable Data: stateful RDF data which embed its own semantic and which doesn’t rely on external (mostly stateless since we can rarely rely on their stated versions) ontologies. My intuition is that it would be possible to serialize RDF data in such a way that it would be self-aware of its own semantic which means that it would know how it can be interpreted, how it can be used, and how it should be validated. The idea is to end-up with Portable Data snippets that could be exchanged between systems without requiring prior, or post, schemas (ontologies) to interpret that information. Then web service endpoints such as OSF, or any other kind of applications, could emit such Portable Data structures without requiring any subsequent ontologies analysis from their part.

However, before being able to implement and demonstrate these intuitions, the first step is to check what such a RDF serialization may looks like. This is the goal of this blog post.

Serializing RDF Data as Clojure Code

Where to start? There are probably multiple ways to do that. Do we want to do that using a map, a structmap, a records, or…? What I wanted to use (at least initially) is a basic data structure that would give me the flexibility I need to serialize RDF data. I wanted a core structure such that existing Clojure developers could easily manipulate it using the existing Clojure functions and techniques that they are used to use.

The collection I choose to start with is the map. This key/value pair structure is ideal for serializing RDF data. It looks like JSON code, but is even simpler since it doesn’t require commas nor colons in its syntax.

The crux of the map structure is that in a map, the keys can be: keywords, symbols, strings, characters, booleans and numbers. The only things it cannot be are regular expressions and the nilvalue. What should be stated here is that symbols can be a lot of different things. They are names for vars, functions, etc.

This opens a World of possibilities to serialize RDF data as Clojure code. In fact, the keys of the map can virtually be anything: and this is just too nice to be true!

What we will investigate in the remaining of this blog posts are different ways to serialize RDF data as Clojure code. These are the initial tests I did to test my intuitions. All of them works, but only the last one really opens-up a World of possibilities and that enables me to implement my early intuitions.

Quick Introduction to RDF Data

RDF is nothing else than a bunch of triples of the form:

  • <subject> <predicate> <object>

Where the <subject> is the thing (resource, record, entity, etc.) being described, where the <predicate> is the property (attribute, etc.) that describes the subject and where the <object> is the value of the predicate which can be a reference to another subject, a literal value, etc.

Each <subject> do have at least a type. A type is nothing else than a class of things which is defined in a RDFS schema or a OWL ontology.

Then if you wire these triples together, you get a directed graph which we often refer to as a datataset. It is as simple as that. However, I won’t state that RDF is necessarily simple, since its expressivity (a double-edged sword) can make things much more complex.

The semantic of the data lies into the <predicate> and the type. It is the predicate and the type that tell us how to interpret, and use, the data. It is what is used to validate the data for example. That is exactly where Clojure, and its map structure, can help us to create this kind of portable data.

As you will see below, the serialization of RDF data as Clojure code looks like the structJSON RDF serialization format developed by Structured Dynamics and used at the core of the Open Semantic Framework. It is not a coincidence since that simple structure has been highly effective to serialize and transmit RDF information between OSF web services and other applications such as OSF for Drupal and other JavaScript applications.

Leveraging Serialization’s Hierarchy to Create Triples

Before jumping into Clojure, let’s take a quick look at a really simple structJSON record. What I want to show you is how triples can be extracted from such a data structure. It is the same principle that will be used to extract triples from the Clojure serialization:

[cc lang=’javascript’ line_numbers=’false’]
[raw] “subject”: [
{
“uri”: “http://dataset1.com/record-a/”,
“predicate”: [
{
“rdfs:type”: “http://umbel.org/umbel/rc/Person”
},
{
“iron:prefLabel”: “Bob”
},
{
“foaf:knows”: {
“uri”: “http://dataset2.com/record-b/”
}
}
}
][/raw]
[/cc]

What we leverage here to extract triples is the hierarchy nature of the serialization. Here the "subject" key introduce an array of objects. Each object has a "uri" key which is the identifier (<subject> of a triple). Then the "predicate" key introduces a series of attributes for that record. Each element of the array is a predicate with the key is the prefixed version of the RDF <predicate>. Then you have a value for each of these predicate keys. If you read the documentation, you will see that you can get to another level called the reification of that triple (don’t confuse with Clojure’s reification mechanism) that is used to define extra information related to a triple statement. That structJSON code would produce the following ntriples:

[cc lang=’text’ line_numbers=’false’]
[raw]http://dataset1.com/record-a/ rdfs:type http://umbel.org/umbel/rc/Person .
http://dataset1.com/record-a/ iron:prefLabel “Bob” .
http://dataset1.com/record-a/ foaf:knows http://dataset2.com/record-b/ .[/raw]
[/cc]

Serializing RDF using Maps and Keywords Keys

The most intuitive way to serialize RDF data as Clojure maps would be to create a map where all the keys are keywords. An initial test would be:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def resource {:uri “http://foo.com/1”
:rdf/type [foaf/Person owl/Thing]
:iron/prefLabel {:value “Fred”
:lang nil
:datatype xsd/string}
:foaf/knows [{:uri “http://foo.com/2”
:rei [{:iron/prefLabel [{:value “Bob”
:lang “en”}
{:value “Robert”
:lang “fr”}]}]}
{:uri “http://foo.com/3”
:rei [:iron/prefLabel “Mike”]}]})[/raw]
[/cc]

What we did here is to define a map with the symbol resource. This map is composed of a series of keys and values where the keys are keywords, and were the values can be strings, vectors or maps. The basic serialization rules are:

  • Each map has a :uri key that define the URI of the resource being described
  • Each key is a namespaced key where the root of the namespace is the prefix of the ontology where the <predicate> or type is defined
  • If the predicate is a owl:DatatypeProperty, then its value can be:
    • A vector with one or multiple map and/or string
    • A map which can have four keys:
      • :value which specify the actual string value
      • :lang which specify the language of that string
      • :datatype which specify the datatype of the string
      • :rei which specify reification statements for the triple
    • A string which is the actual value without any additional information about that Literal
  • If the predicate is a owl:ObjectProperty, then its value can be:
    • A vector with one or multiple map, string and/or symbol
    • A map which can have two keys:
      • :uri which specify the actual URI of the referenced resource
      • :rei which specify reification statements for the triple
    • A string which represent the URI of the resource to be referenced
    • A symbol which represents the URI string of the resource to be referenced

Namespacing Keywords

One of the important notion is that the keywords used as map keys are namespaced. This means that they are defined, and live, in their own namespace. This is an essential requirement for a RDF serialization since we re-use multiple ontologies that may share the same name for some of the predicates and that we don’t want these keywords to clash. That is why that by convention we do create each of these keywords in their respective ontology’s namespace. An ontology namespace is defined as the prefix used to refer to the ontology (for example, the Bibliographic Ontology‘s prefix is bibo, so :bibo/shortTitle would be the key referring to the property http://purl.org/ontology/bibo/shortTitle).

Usage

Now let see how we can work with such a structure in Clojure:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; Return the values of the rdf/type property
(:rdf/type resource)
(resource :rdf/type)
(get resource :rdf/type)

;; Return all the properties that describes the resource
(keys resource)

;; Get the URI of the first person known by Fred
(:uri (first (:foaf/knows resource)))

;; Get the French name of the first person known by Fred
(:value (second (:iron/prefLabel (first (:rei (first (:foaf/knows resource)))))))

;; Update the name of Fred to Frederick
(update-in resource [:iron/prefLabel :value] str “erick”)

;; Output the difference betweeen the original resource and the updated one
(diff resource (update-in resource [iron/prefLabel value] str “erick”))

;; Find the value of a key
(find resource iron/prefLabel)

;; Select values of multiple keys
(select-keys resource [iron/prefLabel foaf/knows])

;; Merge a resource into another resource. The URI and properties of the later resource are kept into the merged resource
(def res-1 {uri “http://foo.com/datasets/test/1”
rdf/type owl/Thing
iron/prefLabel “Preferred Label”})

(def res-2 {uri “http://foo.com/datasets/test/2”
rdf/type owl/Thing
iron/altLabel “Alternative Label”})

(merge res-1 res-2)[/raw]
[/cc]

That is all good and easy. We use Clojure’s core functions and mechanism to easily manipulate RDF data into our application.

However, is this implementing the intuitions I started with? Definitely not. This is more like a conventional serialization format for RDF just like structJSON. The thing here is that if we want to do any kind of validation on this data, if we want the data to be self-aware of its own semantic, then it is not possible when keys are keywords. We would need external mechanisms to create that map structure, then to check what it refers to (the properties, the types, etc.). And then we would have to look them up into their respective ontologies and finally we would have to validate the data structure according to what these ontologies are saying by re-processing that map structure.

This is not quite what I had in mind and what my intuition was telling me.

Serializing RDF using Maps and Symbol Keys

Let push this idea further. What if the keys of the map that represent our RDF data are not keywords, but symbols? Symbols in Clojure name things like vars, functions, etc. Initially, let’s use symbols that refers to the URI (string) of the <predicate> and the types.

The serialization would look like:

[cc lang=’lisp’ line_numbers=’false’]
[raw](def resource {uri “http://foo.com/1”
rdf/type [foaf/Person owl/Thing]
iron/prefLabel {value “Fred”
datatype xsd/string}
foaf/knows [{uri “http://foo.com/2”
rei [{iron/prefLabel [{value “Bob”
lang “en”}
{value “Robert”
lang “fr”}]}]}
{uri “http://foo.com/3”
rei [iron/prefLabel “Mike”]}]})[/raw]
[/cc]

Now our resource is defined with the same structure, except that the keys are actual symbols. In this second iteration, we will consider that the symbols we defined here are representing a string which is the URI of the predicates or the types.

The real advantage of using symbol over keywords for what we are doing with these RDF serialization is that a symbol can:

  • Have a docstring
  • Have meta-data
  • The evaluation of the symbol will results into getting the actual full URI of the predicates/types

These are obvious enhancements over using keywords. First, by being able to define docstrings, which means that we will be able to document these properties and types such that Clojure IDEs can display the documentation of these symbols while you are writing/editing RDF data in Clojure.

Clojure’s meta-data system will be highly leveraged in the final candidate serialization format that I will cover in another blog post, so I won’t discuss it further for the moment.

Finally, once we evaluate such a map, we get the map along with all the evaluated properties/types which refers to their full URI. The evaluation of such as structure [(eval resource)] looks like:

[cc lang=’lisp’ line_numbers=’false’]
[raw]{uri “http://foo.com/1”, “http://www.w3.org/1999/02/22-rdf-syntax-ns#type” [“http://xmlns.com/foaf/0.1/Person” “http://www.w3.org/2002/07/owl#Thing”], “http://purl.org/ontology/iron#prefLabel” {value “Fred”, datatype “http://www.w3.org/2001/XMLSchema#string”}, “http://xmlns.com/foaf/0.1/knows” [{uri “http://foo.com/2”, rei [{“http://purl.org/ontology/iron#prefLabel” [{value “Bob”, lang “en”} {value “Robert”, lang “fr”}]}]} {uri “http://foo.com/3”, rei [“http://purl.org/ontology/iron#prefLabel” “Mike”]}]}[/raw]
[/cc]

As you can see, we can get the full description of this resource with the full expansion of the URIs referenced by the symbols.

The same parsing rules defined in the previous section applies for this new format that uses symbols instead of keys. The same comments regarding namespaces applies here too.

The usage is nearly identical except that a symbol is not a function like the keys which means that you cannot get the value of a key like this when the key is a symbol:

[cc lang=’lisp’ line_numbers=’false’]
[raw](rdf/type resource)[/raw]
[/cc]

What you have to do is to access that using one of the following two methods:

[cc lang=’lisp’ line_numbers=’false’]
[raw](resource rdf/type)
(get resource rdf/type)[/raw]
[/cc]

Even if we improved upon using keywords as keys for the map, we still don’t have any kind of embedded semantic or auto-validation capabilities as my intuition was telling me. It remains the same kind of structure without much significant improvements.

Serializing RDF using Maps and Symbol Keys Referring to Functions

Let’s change our mind, and let evolve this idea of symbols: what if the symbols we define in the map are actually functions instead of strings?

What!?!?

A function could be the key of a map in Clojure?

Well not directly, but yes. In Clojure symbols are naming different things such as functions. This is quite an important feature of Clojure: it makes the distinction between how things are named, and these actual things.

This means that what is really used as keys in our map structure is a symbol. However, that symbol happen to refer to a function. So it is not the function itself that is used as a key, but the actual thing that refers to it which is the symbol.

However, the result is the same: if we evaluate the map, we will get a series of symbols that evaluates to functions. That is exactly what we were looking for: that little gem, hanging around, just waiting to be picked-up.

This opens an overwhelming number of possibilities. This means that we have a data structure that can be evaluated to a series of functions and that can be executed. That is exactly what should enable us to define that Portable [RDF] Data serialization format.

That means that we won’t only be able to define RDF triples as Clojure code, but that we could even execute that Clojure code to do different things with the data, such as auto-validating itself, etc.

Finally, what if we consider RDF predicates as Clojure functions? Predicates have all kind of properties and semantics. They can be specified to be used to describe only certain kind of resources, or to refer to specific type of values. Predicates can be symmetric, functional, transitive, etc. What if we simply implement these characterics as Clojure functions? This is what this whole thing is mean to be. When evaluating and “running” that RDF map structure, we would simply execute these functions that define the semantic and characteristics of these predicates. That is exactly where lies my intuitions: we would end-up with a RDF serialization format that “embed” it own semantic and that can be used to self-validate itself by executing the structure. That is what I would refer to as Portable Data: stateful data with embedded stateful semantic.

The initial version of this other revision of the RDF serialization as Clojure code will be outlined in the next blog post since its discussion warrant a full blog post in itself. However I think that you can start understanding where I am heading with these intuitions and why I am using Clojure to test them.

Once an initial version of this serialization will be outlined, we will see how it can be used, what are the benefits, how the idea of Portable Data could be leveraged, how it can help creating and managing data using traditional IDEs such as Emacs. Once the basis will be outlined, we will have all the leisure to explore the benefits of this concept.

Data as Code. Code as Data: Tighther Semantic Web Development Using Clojure

LhrMyRXKX9w!v!gOqzkEBlYSdf8I have been professionally working in the field of the Semantic Web for more than 7 years now. I have been developing all kind of Ontologies. I have been integrating all kind of datasets from various sources. I have been working with all kind of tools and technologies using all kind of technologies stacks. I have been developing services and user interfaces of all kinds. I have been developing a set of 27 web services packaged as the Open Semantic Framework and re-implemented the core Drupal modules to work with RDF data has I wanted it to. I did write hundred of thousands of line of codes with one goal in mind: leveraging the ideas and concepts of the Semantic Web to make me, other developers, ontologists and data-scientists working more accurately and efficiently with any kind data.

However, even after doing all that, I was still feeling a void: a disconnection between how I was thinking about data and how I was manipulating it using the programming languages I was using, the libraries I was leveraging and the web services that I was developing. Everything is working, and is working really well; I did gain a lot of productivity in all these years. However, I was still feeling that void, that disconnection between the data and the programming language.

Every time I want to work with data, I have to get that data serialized using some format, then I have to parse it using a parser available in the language I am working with. Then the data needs to be converted into an internal structure by the parser. Then I have to use all kind of specialized APIs to work with the data represented by that structure. Then if I want to validate the data that I am working with, I will probably have to use another library that will perform the validation for me which may force me to migrate that data to another system that will make it available to these reasoners and validators. Etc, etc, etc…

All this is working: I have been doing this for years. However, the level of interaction between all these systems is big and the integration take time and resources. Is there a way to do things differently?

The Pink Book

417XBWM48NL._Once I realized that, I started a quest to try to change that situation. I had no idea where I was heading, and what I would find, but I had to change my mind, to change my view-point, to start getting influenced by new ideas and concepts.

What I realized is how disconnected mainstream programming languages may be with the data I was working with. That makes a natural first step to start my investigation. I turned my chair and started to stare at my bookshelves. Then, like the One Ring, there was this little Pink (really pink) book that was staring at me: Lambda-calcul types et mod[raw]è[/raw]les. I bought that books probably 10 years ago, then I forgot about it. I always found its cover page weird, and its color awkward. But, because of these uncommon features, I got attracted by it.

Re-reading about lambda-calculus opened my eyes. It leaded me to have a particular interest in homoiconic programming languages such as Lisp and some of its dialects.

Code as Data. Data as Code.

Is this not what I was looking for? Could this not fill the void I was feeling? Is this not where my intuition was heading?

What if the “data” I manipulate is the same as the code I am writing? What if the data that I publish could be the code of a module of an application? What if writing code is no different than creating data? What if data could be self-aware of its own semantic? What if by evaluating data structures, I would validate that data at the same time? What if “parsing” my data is in fact evaluating the code of my application? What if I could reuse the tools and IDEs I use for programming, but for creating, editing and validating data? Won’t all these things make things simpler and make me even more productive to work with data?

My intuition tells me: yes!

We have a saying at Structured Dynamics: the right tool for the right job.

That seems to be the kind of tool I need to fill that void I was feeling. I had the feeling that the distinction between the code and the data should be as minimal as possible and homoiconic languages seems to be the right tool for that job.

Code as Data. Data as Code.

That is all good, but what does that really mean? What are the advantages and benefits?

That is the starting of a journey, and this is what we will discover in the coming weeks and months. Structured Dynamics is starting to invest resources into that new project. We choose to do our work using Clojure instead of other Lisp dialects such as Common Lisp. We choose Clojure for many reason: it is compiled in JVM bytecode. This means that you can re-use any of this code into any other Java applications and this also means that you can re-use any Java libraries natively into Clojure. But we also did use it because of its native way to handle concurrency and parallelism, its unique way to manage metadata within data structures, for its meta-programming capabilities using its macro system that enable us to create DSL, etc.

The goal was to create a new serialization format for RDF and to serialize RDF data as Clojure code. The intuition was that RDF data would then become an integral part of Clojure applications because the data would be the code as well.

The data would be self-aware of its own semantic, which means that by evaluating the Clojure “RDF” code it would also auto-validate itself using its embedded semantic. The RDF data would be in itself an [Clojure] application that would be self-aware of its own semantic and that would know how to validate itself.

That is the crux of my thinking. Then, how could this be implemented?

That is what I will cover in the coming weeks and months. We choose to use Clojure because it seems to be a perfect fit for that job. We will discover the reasons over time. However, the goal of these blog posts is to show how RDF can be serialized into [Clojure] code and the benefits of doing so. It is not about showing all the neat features of, and the wonderful minding behind Clojure. For that, I would strongly suggest you to get started with Clojure by reading the material covered in Tips for Clojure Beginners, and particularly to take a few hours to listen Rich Hickey’s great videos.

 

 

My Optimal GNU Emacs Settings for Developing Clojure (so far)

Note: this blog post has been revised with this other blog post.

In the coming months, I will start to publish a series of blog posts that will explain how RDF data can be serialized in Clojure code and more importantly what are the benefits of doing this. At Structured Dynamics, we started to invest resources into this research project and we believe that it will become a game changer regarding how people will consume, use and produce RDF data.

But I want to take a humble first step into this journey just by explaining how I ended up configuring Emacs for working with Clojure. I want to take the time to do this since this is a trials and errors process, and that it may be somewhat time-consuming for the new comers.

Light Table

Before discussing how I configured Emacs, I want to introduce you to the new IDE: Light Table. This new IDE is mean to be the next generation of code editor. If you are new to Clojure, and more particularly if you never used Emacs before, I would strongly suggest you to start with this code editor. It is not only simple to use, but all the packages you will require to work with Clojure are already built-in.

As you may know, GNU Emacs has been developed using Emacs Lisp (a Lisp dialect). This means that it can be extended by installing and enabling  packages, all configurations options and behaviors can be changes, and even while it is running! Light Table is no different. It has been developed in ClosureScript, and it can be extended the same way. To me, the two real innovations with Light Table are:

  • The instarepl
  • The watches

The instarepl is a way to evaluate the value of anything, while you are coding, directly inline in your code. This is really powerful and handy when prototyping and testing code. Every time you type some code, it get evaluated in the REPL, and displayed inline in the code editor.

The watches are like permanent instarepl that you place within the code. Then every time the value changes, you see the result in the watch section. This is really handy when you have to see the value of some computation while the application, or part of the application, are running. You get a live output of what is being computed, directly into your code.

The only drawback I have with LightTable is that there is no legacy REPL available (yet?). This means that if you want to evaluate something unrelated to your code, you have to write the code directly into the editor and then evaluate it with the instarepl. Another issue regarding some use cases is that the evaluation of the code can become confusing like when you define a Jetty server in your code. Since everything get evaluated automatically (if the live mode is enabled) then it can start the server without you knowing it. Then to stop it, you have to write a line of code into your code and then to evaluate it to stop the server.

Because of the nature of my work, I am a heavy user of multiple monitors (daily working with six monitors). This means that properly handling multiple monitors is essential to my productivity. That is another issue I have with LightTable: you can create new windows that you can move to other monitors, but these windows are unconnected: they are different instances of LightTable.

Simple is beautiful, and it is why I really do like LightTable and why I think it is what beginners should use to start working with Clojure. However, it is not yet perfect for what I have to do. That is why I choose to use GNU Emacs for my daily work.

GNU Emacs

I don’t think that GNU Emacs needs any kind of introduction. It is heavy, it is unnatural, it takes time to get used to, the learning curve is steep, but… hell it is powerful for working with Lisp dialects like Clojure!

The problem with Emacs is not just to learn the endless list of key bindings (even if you can go a long way with the core ones), but also to configure it for your taste. Since everything can be configured, and that there exists hundred of all kind of packages, it takes time to configures all the options you want, and all the modules you require. This is the main reason I wrote this blog post: to share my (currently) best set of configuration options and packages for using Emacs for developing with Clojure.

I am personally developing on Windows 8, but these steps should be platform agnostic. You only have to download and install the latest GNU Emacs 24 version.

The first thing you have to do is to locate you .emacs file. All the configurations I am defining in this blog post goes into that file.

Packages

Once Emacs is installed, the first thing you have to do is to install all the packages that are required to develop in Clojure or that will make your life easier for handling the code. The packages that you have to install are:

  • cider
    • Clojure Integrated Development Environment and REPL – This is like Slime for Common Lisp. This is what turns Emacs into a Clojure IDE
    • Important note: make sure that the Cider version you are installing is coming from the MELPA-Stable repository, and not the MELPA one. At the time of the publication of the blogpost, the latest stable release is 0.6.
  • clojure-mode
    • Major mode for Clojure code
  • clojure-test-mode
    • Minor mode for Clojure tests
  • auto-complete
    • Auto Completion for GNU Emacs – This is what is used to have auto-completion capabilities into your code and in the mini-buffer
  • ac-nrepl
    • Auto-complete sources for Clojure using nrepl completions – This is what is used to add auto-completion capabilities to the NREPL
  • paredit
    • minor mode for editing parentheses  -*- Mode: Emacs-Lisp -*- – This is what will do all the Lisp like code formatting (helping you managing all these parenthesis)
  • popup
    • Visual Popup User Interface – This is what will enable popup contextual menus when using auto-completion in your code and in the NREPL
  • raindow-delimiters
    • Highlight nested parens, brackets, braces a different color at each depth – This is really handy to visually see where you are with your parenthesis. An essential to have
  • rainbow-mode
    • Colorize color names in buffers

Before installing them, we have to tell Emacs to use the Marmelade packages repository where all these packages are hosted and ready to the installed into your Emacs instance. At the top of your .emacs file, put:

[cc lang=’lisp’ line_numbers=’false’]
[raw](require ‘package)

(add-to-list ‘package-archives
‘(“melpa-stable” . “http://melpa-stable.milkbox.net/packages/”))

(add-to-list ‘package-archives
‘(“melpa” . “http://melpa.milkbox.net/packages/”))

(add-to-list ‘package-archives
‘(“marmalade” . “http://marmalade-repo.org/packages/”))

;; Initialize all the ELPA packages (what is installed using the packages commands)
(package-initialize)[/raw]
[/cc]

Important note: only use the MELPA repository if you want to install non-stable modules such as the Noctulix theme. If you are not expecting using it, then I would strongly suggest you to remove it and only to keep the MELPA-Stable repository in that list.

If you are editing your .emacs file directly into Emacs, and you can re-evaluate the settings file using Emacs, then by moving cursor at each top-level expression end (after closing parenthesis) and press C-x C-e. However, it may be faster just to close and restart Emacs to take the new settings into account. You can use any of these methods for the following set of settings changes.

Before changing any more settings, we will first install all the required packages using the following sequence of commands:

  • M-x package-install [RET] cider [RET]
  • M-x package-install [RET] clojure-mode [RET]
  • M-x package-install [RET] clojure-test-mode[RET]
  • M-x package-install [RET] auto-complete[RET]
  • M-x package-install [RET] ac-nrepl [RET]
  • M-x package-install [RET] paredit[RET]
  • M-x package-install [RET] popup [RET]
  • M-x package-install [RET] rainbow-delimiters [RET]
  • M-x package-install [RET] rainbow-mode [RET]

Additionally, you could have used M-x package-list-packages, then move your cursor in the buffer to the packages’ line. Then press i (for install) and once all the packages are selected, you could have press x (execute) to install all the packages all at once.

In the list of commands above, M-x is the “meta-key” normally bound to the left Alt key on your keyboard. So, M-x usually means Alt-x.

Now that all the packages are installed, let’s take a look at how we should configure them.

Configuring Keyboard

If you are using an English/US keyboard, you can skip this section. Since I use a French Canadian layout (On an English/US Das Keyboard!), I had multiple issues to have my keys working since all the binding changed in Emacs. To solve this problem, I simply had to define that language configuration option. Then I had to start using the right Alt key of my keyboard to write my brackets, curly brackets, etc:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; Enable my Canadian French keyboard layout
(require ‘iso-transl)[/raw]
[/cc]

Configuring Fonts

Since I am growing older (and that I have much screen estates with six monitors), I need bigger fonts. I like coding using Courier New, so I just configured it to use the font size 13 instead of the default 10:
[cc lang=’lisp’ line_numbers=’false’]
[raw];; Set bigger fonts
(set-default-font “Courier New-13”)[/raw]
[/cc]

Cider and nREPL

The next step is to configure Cider and the nREPL which are the two pieces that turns Emacs into a wonderful Clojure IDE:

[cc lang=’lisp’ line_numbers=’false’]
[raw](add-hook ‘clojure-mode-hook ‘turn-on-eldoc-mode)
(setq nrepl-popup-stacktraces nil)
(add-to-list ‘same-window-buffer-names “nrepl“)[/raw]
[/cc]

Auto-completion

The next step is to configure the auto-completion feature everywhere in Emacs: in any buffer, nREPL or in the mini-buffer. Then we want the auto-completion to appear in a contextual menu where the docstrings (documentation) of the functions will be displayed:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; General Auto-Complete
(require ‘auto-complete-config)
(setq ac-delay 0.0)
(setq ac-quick-help-delay 0.5)
(ac-config-default)

;; ac-nrepl (Auto-complete for the nREPL)
(require ‘ac-nrepl)
(add-hook ‘cider-mode-hook ‘ac-nrepl-setup)
(add-hook ‘cider-repl-mode-hook ‘ac-nrepl-setup)
(add-to-list ‘ac-modes ‘cider-mode)
(add-to-list ‘ac-modes ‘cider-repl-mode)[/raw]
[/cc]

Popping Contextual Documentation At Any Time

What is really helpful is to be able to pop the documentation for any symbol at any time just by pressing a series of keys. What need to be done is to configure Cider & ac-nrepl to bind this behavior to the C-c C-d sequence of keys:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; Poping-up contextual documentation
(eval-after-load “cider”
‘(define-key cider-mode-map (kbd “C-c C-d”) ‘ac-nrepl-popup-doc))[/raw]
[/cc]

Par Edit

Par Edit is the package that will help you out automatically formatting you Clojure code. It will balance the parenthesis, automatically indenting your S-expressions, etc.

[cc lang=’lisp’ line_numbers=’false’]
[raw](add-hook ‘clojure-mode-hook ‘paredit-mode)[/raw]
[/cc]

Show Parenthesis Mode

Another handy feature is to enable, by default, the show-parent-mode configuration option. That way, every time the cursor points to a parenthesis, the parent parenthesis will be highlighted into the user interface. This is an essential most-have with Par Edit:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; Show parenthesis mode
(show-paren-mode 1)[/raw]
[/cc]

Rainbow Delimiters

Another essential package to have to help you out maintaining these parenthesis. The rainbow delimiters will change the color of the parenthesis depending on how “deep” they are into the structure. Another essential visual cue:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; rainbow delimiters
(global-rainbow-delimiters-mode)[/raw]
[/cc]

Noctilux Theme

Did I say that I like LightTable? In fact, I really to like their dark theme. It is the best I saw so far. I never used any in my life since I never liked any of them. But that one is really neat, particularly to help visualizing Clojure code. That is why I really wanted to get a LightTable theme for Emacs. It exists and it is called Noctilux and works exactly the same way with the same colors.

If you want to install it, you can get it directly from the packages archives. Type M-x package-list-packages then search and install noctilux-theme.

Then enable it by adding this setting:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; Noctilus Theme
(load-theme ‘noctilux t)[/raw]
[/cc]

Binding Some Keys

Then I wanted to bind some behaviors to the F-keys. What I wanted is to be able to run Cider, to be able to start and stop Par Edit and to switch frames (windows within monitors) in a single click. I also added a shortkey for starting speedbar for the current buffer, it is an essential for managing project files. What I did is to bind these behaviors to these keys:

[cc lang=’lisp’ line_numbers=’false’]
[raw](global-set-key [f8] ‘other-frame)
(global-set-key [f7] ‘paredit-mode)
(global-set-key [f9] ‘cider-jack-in)
(global-set-key [f11] ‘speedbar)[/raw]
[/cc]

Fixing the Scroll

There is one thing that I really didn’t like, and it was the default behavior of the scrolling of Emacs on Windows. After some searching, I found the following configurations that I could fix to have a smoother scrolling behavior on Windows:

[cc lang=’lisp’ line_numbers=’false’]
[raw];; scroll one line at a time (less “jumpy” than defaults)

(setq mouse-wheel-scroll-amount ‘(1 ((shift) . 1))) ;; one line at a time

(setq mouse-wheel-progressive-speed nil) ;; don’t accelerate scrolling

(setq mouse-wheel-follow-mouse ‘t) ;; scroll window under mouse

(setq scroll-step 1) ;; keyboard scroll one line at a time[/raw]
[/cc]

Complete Configuration File

Here is the full configuration file that I am using:

[cc lang=’lisp’ line_numbers=’false’]
[raw](require ‘package)

(add-to-list ‘package-archives
‘(“melpa-stable” . “http://melpa-stable.milkbox.net/packages/”))

(add-to-list ‘package-archives
‘(“melpa” . “http://melpa.milkbox.net/packages/”))

(add-to-list ‘package-archives
‘(“marmalade” . “http://marmalade-repo.org/packages/”))

;; Initialize all the ELPA packages (what is installed using the packages commands)
(package-initialize)

;; Enable my Canadian French keyboard layout
(require ‘iso-transl)

;; Set bigger fonts
(set-default-font “Courier New-13”)

;; Cider & nREPL
(add-hook ‘clojure-mode-hook ‘turn-on-eldoc-mode)
(setq nrepl-popup-stacktraces nil)
(add-to-list ‘same-window-buffer-names “nrepl“)

;; General Auto-Complete
(require ‘auto-complete-config)
(setq ac-delay 0.0)
(setq ac-quick-help-delay 0.5)
(ac-config-default)

;; ac-nrepl (Auto-complete for the nREPL)
(require ‘ac-nrepl)
(add-hook ‘cider-mode-hook ‘ac-nrepl-setup)
(add-hook ‘cider-repl-mode-hook ‘ac-nrepl-setup)
(add-to-list ‘ac-modes ‘cider-mode)
(add-to-list ‘ac-modes ‘cider-repl-mode)

;; Popping-up contextual documentation
(eval-after-load “cider”
‘(define-key cider-mode-map (kbd “C-c C-d”) ‘ac-nrepl-popup-doc))

;; paredit
(add-hook ‘clojure-mode-hook ‘paredit-mode)

;; Show parenthesis mode
(show-paren-mode 1)

;; rainbow delimiters
(global-rainbow-delimiters-mode)

;; Noctilus Theme
(load-theme ‘noctilux t)

;; Switch frame using F8
(global-set-key [f8] ‘other-frame)
(global-set-key [f7] ‘paredit-mode)
(global-set-key [f9] ‘cider-jack-in)
(global-set-key [f11] ‘speedbar)

;; scroll one line at a time (less “jumpy” than defaults)
(setq mouse-wheel-scroll-amount ‘(1 ((shift) . 1))) ;; one line at a time
(setq mouse-wheel-progressive-speed nil) ;; don’t accelerate scrolling
(setq mouse-wheel-follow-mouse ‘t) ;; scroll window under mouse
(setq scroll-step 1) ;; keyboard scroll one line at a time[/raw]
[/cc]

Conclusion

Now that we have the proper development environment in place, the next blog posts will really get into the heart of the matter: what are the different ways to serialize RDF data in Clojure code, how the generated code can be used, what are the benefits, how it changes the way that data (RDF in this case, but really any data) can be produced and consumed.

We think that there are profound implications into how we, as Semantic Web specialists, will work with data instances and ontologies in the future. The initial project that will embed and benefit from these new principles and techniques will be the next version of the UMBEL ontology.

Final note: there are an endless list of features and packages for Emacs. Obviously, I don’t know all of them, so if you are aware of any settings or packages that I missed here and that could improve this setup, please share them in the comments.

jQuery Cookie Pluging Extended With HTML5 localStorage And Chunked Cookies

Is there a web developer that never used cookies to save some information in a user’s browser? There may be, but they should be legion. As you probably know, the problem with cookies is that their implementation in browsers is random: some will limit the size of the cookie to 4096 bytes, others will limit the number of cookies from a specific domain to 50, others will have no perceivable limits, etc.


In any case, if one of these limits is reached, the cookie is simply not created the browser. This is fine, because web developer expects cookies to fail from time to time, and the system they develop has to cope with this unreliableness. However, this situation can sometimes become frustrating, and it is why I wanted to extend the default behavior of the jQuery Cookie plugin with a few more capabilities.

This extension to the jQuery Cookie plugin adds the capability to save content that is bigger than 4096 bytes long using two different mechanism: the usage of HTML5’s localStorage, or the usage of a series of cookies where the content is chunked and saved. This extension is backward compatible with the jQuery Cookie plugin and its usage should be transparent to the users. Even if existing cookies have been created with the normal Cookie plugin, they will still be usable by this new extension. The usage syntax is the same, but 3 new options have been created.

Now, let’s see how this plugin works, how developers should use it, what are its limitations, etc.

You can immediately download the jQuery Extended Cookie plugin from here:

Limitations Of Cookies

First, let’s see what the RFC 2109 says about the limitations of cookies in web browsers. Browsers should normally have these implementation limits (see section 6.3):

   Practical user agent implementations have limits on the number and
   size of cookies that they can store.  In general, user agents' cookie
   support should have no fixed limits.  They should strive to store as
   many frequently-used cookies as possible.  Furthermore, general-use
   user agents should provide each of the following minimum capabilities
   individually, although not necessarily simultaneously:

      * at least 300 cookies
      * at least 4096 bytes per cookie (as measured by the size of the
        characters that comprise the cookie non-terminal in the syntax
        description of the Set-Cookie header)
      * at least 20 cookies per unique host or domain name

   User agents created for specific purposes or for limited-capacity
   devices should provide at least 20 cookies of 4096 bytes, to ensure
   that the user can interact with a session-based origin server.

   The information in a Set-Cookie response header must be retained in
   its entirety.  If for some reason there is inadequate space to store
   the cookie, it must be discarded, not truncated.

   Applications should use as few and as small cookies as possible, and
   they should cope gracefully with the loss of a cookie.

New Options

Before I explains how this extension works, let me introduce three new options that have been added to the Cookie plugin. These new options will be put into context, and properly defined later in this blog post.

  • maxChunkSize – This defines the maximum number of bytes that can be saved in a single cookie. (default: 3000)
  • maxNumberOfCookies - This is the maximum number of cookies that can be created for a single domain name. (default: 20)
  • useLocalStorage – This tells the extended Cookie plugin to use the HTML5’s localStorage capabilities of the browser instead of a cookie to save that value. (default: true)

How Does This Extension Works?

As I said in the introduction of this blog post, this extension to the jQuery Cookie plugin does two things:

  1. It uses the HTML5 localStorage capabilities of the browser if this feature is available instead of relying on the cookies. However, if cookies are needed by the developer, this feature can be turned off with the useLocalStorage = false option
  2. If the localStorage option is disable, or simply not available on a browser, and if the content is bigger than the limit of the size of a cookie, then this extension will chunk the input content, and save it in multiple cookies

If the useLocalStorage is true, then the plugin will try to see if the HTML5 localStorage mechanism is available on the browser. If it is, then it will use that local storage to save and retrieve content to the browser. If it is not, then the plugin will act like if useLocalStorage is false and the process will continue by using cookies to save and read that content from the browser.

If useLocalStorage is false, or if the HTML5 localStorage mechanism is not available on the browser, then the plugin will check if the content is bigger than the maxChunkSize option, than all the chunks will be saved in different cookies until it reaches the limit imposed by the maxNumberOfCookies option.

If cookies are used, then two use-cases can happen:

  1. The content is smaller than or equal to maxChunkSize
  2. The content is bigger than maxChunkSize

If the content is smaller than or equal to maxChunkSize than only one cookie will be created by the browser. The name of the cookie will be the value provided to the key parameter.

If the content is bigger than maxChunkSize than multiple cookies will be created, one per chunk. The convention is that the name of the first cookie is the value provided to the key parameter. The name of the other chunks is the value provided to the key parameter with the chunk indicator ---ChunkNum append to it. For example, if we have a cookie with a content of 10000 bytes that has maxChunkSize defined to 4000 bytes, then these three cookies would be created:

  • cookie-name
  • cookie-name---1
  • cookie-name---2

Usage

Now, let’s see how this extended jQuery Cookie plugin should be used in your code. The usage of the extension is no different from the usage of the normal jQuery Cookie plugin. However, I am showing how to use the new options along with how to use the plugin in general.

Create a Cookie

Let’s create a cookie that expires in 365 days and where the path is the root:

[cc lang=’javascript’ ]

$.cookie(‘my-cookie’, “the-content-of-my-cookie”, { expires: 365, path: “/” });

[/cc]

By default, this value will be persisted in the localStorage if the browser supports it, and not in a cookie. So, let’s see how to force the plugin to save the content in a cookie by using the useLocalStorage option:

[cc lang=’javascript’ ]

$.cookie(‘my-cookie’, “the-content-of-my-cookie”, {useLocalStorage: false, expires: 365, path: “/” });

[/cc]

Delete a Cookie

Let’s see how a cookie can be deleted. The method is simply to put null as the value of the cookie. This will instruct the plugin to remove the cookie.

[cc lang=’javascript’ ]

$.cookie(‘my-cookie’, null);

[/cc]

With that call, the plugin will try to remove my-cookie both in the localStorage and in the cookies.

Read a Cookie

Let’s see how we can read the content of a cookie:

[cc lang=’javascript’ ]

var value = $.cookie(‘my-cookie’);

[/cc]

With this call, value will get the content that has been saved in the localStorage, or the cookies. This will depend if the localStorage was available in the browser.

Now, let’s see how to force reading the cookies by bypassing the localStorage mechanism:

[cc lang=’javascript’ ]

var value = $.cookie(‘my-cookie’, {useLocalStorage: false});

[/cc]

Note that if the cookie is not existing for a key, then the $.cookie() function will return null.

Using Limitations

Let’s see how to use the maxNumberOfCookies and maxChunkSize options to limit the size and the number of cookies to be created.

With this example, the content will be saved in multiple cookies of 1000 bytes each up to 30 cookies:

[cc lang=’javascript’ ]

var value = $.cookie(‘my-cookie’, “the-content-of-my-cookie-is-10000-bytes-long…”, {useLocalStorage: false, maxChunkSize  = 1000, maxNumberOfCookies = 30, expires: 365, path: “/” });

[/cc]

Limitations

Users have to be aware of the limitations of this enhanced plugin. Depending on the browser, the values of the maxChunkSize and the maxNumberOfCookies options should be different. In the worse case, some cookies (or cookies chunks) may simply not be created by the browser. As stated in the RFC 2109, the web applications have to take that fact into account, and be able to gracefully cope with this.

Future Enhancements

In the future, this extension should detect the browser where it runs, and setup the maxChunkSize and the maxNumberOfCookies parameters automatically depending on the cookies limitation of each browser.

Conclusion

I had to create this extension to the jQuery Cookie plugin to be able to store the resultsets returned by some web service endpoints. It is only used to limit the number of queries sent to these endpoints. Since the values returned by the endpoints are nearly static, that they are loaded at each page view and that they are a few kilobytes big, I had to find a way to save that information in the browser, and to overcome the size limitation of the cookies if possible. I also needed to be able to cope with older versions of browsers that only supports cookies. In the worse case scenario, the browser will simply send the request to the endpoints at each page load for the special use-cases where nothing works: not the cookies and not the localStorage. But at least, my application will benefit of this enhancement from the 95% of the users were one of these solutions works.