New Mapping Semantic Component In JavaScript

 

I am please to announce the release of the new sWebMap Semantic Component in JavaScript. This new mapping component is a standalone JavaScript application that can be integrated on any new or existing web sites and that interact with an Open Semantic Framework (OSF) instance to search, browse, filter and display with geographically-located information on an interactive map.

Features

The sWebMap is a rich mapping tool that can easily be integrated on any webpage, and that can be extensively customized. The sWebMap does support these features:

  • Full text search for searching and displaying results on a map
  • Extensive filtering capabilities
    • Filtering by dataset source
    • Filtering by type
    • Filtering by attribute/value
    • Filtering of records that belongs to a specific geographic region
  • Display of record on the map using:
    • Different markers depending on the type of record to display (determined by the ontologies)
    • Polygon shapes for records that refers to a geographic region
    • Polyline shapes for records that refers to a geographically-located path
  • Templating of records in a resultset depending on their type
  • Templating of records’ preview, displayed in an overlay window, depending on their type
  • Persist records on the map accros searches and filtering operations
  • Supports map sessions
    • Save map sessions
    • Load saved map sessions
    • Delete saved map sessions
    • Share saved map sessions
  • Supports a multiple-maps mode
    • Three focus maps are available under the main map
    • Each map focus on a particular region of the main map
    • User can switch between focus map to see different records in different region

 

Normal Mode

Here is what the default sWebMap, in normal mode, using a few datasets related to the city of Iowa looks like. You can also interact with this sWebMap instance directly on the Citizen DAN demo website here.


Multiple Windows Mode

Here is what the default sWebMap, in multiple windows mode, using a few datasets related to the city of Iowa looks like. You can also interact with this sWebMap instance directly on the Citizen DAN demo website here.

 


 

Under the Hood: The Open Semantic Framework

Each sWebMap component communicates with an OSF (Open Semantic Framework) instance. More specifically, a sWebMap component will send Search/Filtering queries to a geo-enabled structWSF Search web service endpoint.

Depending on the options you had specified when you created the sWebMap control, each time you move (option), zoom (option) or change the filtering criterias, this will send a query to the Search endpoint. The sWebMap control then requests JSON formatted resultset and display the results to the user.

This means that to implement the sWebMap component on your website, you will need to have:

Download

You can immediately download the entire code source from this GitHub reposiroty:

Installation

Installing the sWebMap component is really easy. In fact, you only have to load a few JavaScript and CSS files, to defined a <div></div> container for the map, and to create a sWebMap component object, which is a single line of code.

Additionally, you can initialize the sWebMap component with one of the multiple options available.

Refer you to the Usage section of the sWebMap component to know exactly how to install and setup a sWebMap component instance.

Resources

Here are some additional resources related to the sWebMap component:

 

jQuery Cookie Pluging Extended With HTML5 localStorage And Chunked Cookies

Is there a web developer that never used cookies to save some information in a user’s browser? There may be, but they should be legion. As you probably know, the problem with cookies is that their implementation in browsers is random: some will limit the size of the cookie to 4096 bytes, others will limit the number of cookies from a specific domain to 50, others will have no perceivable limits, etc.


In any case, if one of these limits is reached, the cookie is simply not created the browser. This is fine, because web developer expects cookies to fail from time to time, and the system they develop has to cope with this unreliableness. However, this situation can sometimes become frustrating, and it is why I wanted to extend the default behavior of the jQuery Cookie plugin with a few more capabilities.

This extension to the jQuery Cookie plugin adds the capability to save content that is bigger than 4096 bytes long using two different mechanism: the usage of HTML5′s localStorage, or the usage of a series of cookies where the content is chunked and saved. This extension is backward compatible with the jQuery Cookie plugin and its usage should be transparent to the users. Even if existing cookies have been created with the normal Cookie plugin, they will still be usable by this new extension. The usage syntax is the same, but 3 new options have been created.

Now, let’s see how this plugin works, how developers should use it, what are its limitations, etc.

You can immediately download the jQuery Extended Cookie plugin from here:

Limitations Of Cookies

First, let’s see what the RFC 2109 says about the limitations of cookies in web browsers. Browsers should normally have these implementation limits (see section 6.3):

   Practical user agent implementations have limits on the number and
   size of cookies that they can store.  In general, user agents' cookie
   support should have no fixed limits.  They should strive to store as
   many frequently-used cookies as possible.  Furthermore, general-use
   user agents should provide each of the following minimum capabilities
   individually, although not necessarily simultaneously:

      * at least 300 cookies
      * at least 4096 bytes per cookie (as measured by the size of the
        characters that comprise the cookie non-terminal in the syntax
        description of the Set-Cookie header)
      * at least 20 cookies per unique host or domain name

   User agents created for specific purposes or for limited-capacity
   devices should provide at least 20 cookies of 4096 bytes, to ensure
   that the user can interact with a session-based origin server.

   The information in a Set-Cookie response header must be retained in
   its entirety.  If for some reason there is inadequate space to store
   the cookie, it must be discarded, not truncated.

   Applications should use as few and as small cookies as possible, and
   they should cope gracefully with the loss of a cookie.

New Options

Before I explains how this extension works, let me introduce three new options that have been added to the Cookie plugin. These new options will be put into context, and properly defined later in this blog post.

  • maxChunkSize - This defines the maximum number of bytes that can be saved in a single cookie. (default: 3000)
  • maxNumberOfCookies - This is the maximum number of cookies that can be created for a single domain name. (default: 20)
  • useLocalStorage - This tells the extended Cookie plugin to use the HTML5′s localStorage capabilities of the browser instead of a cookie to save that value. (default: true)

How Does This Extension Works?

As I said in the introduction of this blog post, this extension to the jQuery Cookie plugin does two things:

  1. It uses the HTML5 localStorage capabilities of the browser if this feature is available instead of relying on the cookies. However, if cookies are needed by the developer, this feature can be turned off with the useLocalStorage = false option
  2. If the localStorage option is disable, or simply not available on a browser, and if the content is bigger than the limit of the size of a cookie, then this extension will chunk the input content, and save it in multiple cookies

If the useLocalStorage is true, then the plugin will try to see if the HTML5 localStorage mechanism is available on the browser. If it is, then it will use that local storage to save and retrieve content to the browser. If it is not, then the plugin will act like if useLocalStorage is false and the process will continue by using cookies to save and read that content from the browser.

If useLocalStorage is false, or if the HTML5 localStorage mechanism is not available on the browser, then the plugin will check if the content is bigger than the maxChunkSize option, than all the chunks will be saved in different cookies until it reaches the limit imposed by the maxNumberOfCookies option.

If cookies are used, then two use-cases can happen:

  1. The content is smaller than or equal to maxChunkSize
  2. The content is bigger than maxChunkSize

If the content is smaller than or equal to maxChunkSize than only one cookie will be created by the browser. The name of the cookie will be the value provided to the key parameter.

If the content is bigger than maxChunkSize than multiple cookies will be created, one per chunk. The convention is that the name of the first cookie is the value provided to the key parameter. The name of the other chunks is the value provided to the key parameter with the chunk indicator ---ChunkNum append to it. For example, if we have a cookie with a content of 10000 bytes that has maxChunkSize defined to 4000 bytes, then these three cookies would be created:

  • cookie-name
  • cookie-name---1
  • cookie-name---2

Usage

Now, let’s see how this extended jQuery Cookie plugin should be used in your code. The usage of the extension is no different from the usage of the normal jQuery Cookie plugin. However, I am showing how to use the new options along with how to use the plugin in general.

Create a Cookie

Let’s create a cookie that expires in 365 days and where the path is the root:

$.cookie('my-cookie', "the-content-of-my-cookie", { expires: 365, path: "/" });

By default, this value will be persisted in the localStorage if the browser supports it, and not in a cookie. So, let’s see how to force the plugin to save the content in a cookie by using the useLocalStorage option:

$.cookie('my-cookie', "the-content-of-my-cookie", {useLocalStorage: false, expires: 365, path: "/" });

Delete a Cookie

Let’s see how a cookie can be deleted. The method is simply to put null as the value of the cookie. This will instruct the plugin to remove the cookie.

$.cookie('my-cookie', null);

With that call, the plugin will try to remove my-cookie both in the localStorage and in the cookies.

Read a Cookie

Let’s see how we can read the content of a cookie:

var value = $.cookie('my-cookie');

With this call, value will get the content that has been saved in the localStorage, or the cookies. This will depend if the localStorage was available in the browser.

Now, let’s see how to force reading the cookies by bypassing the localStorage mechanism:

var value = $.cookie('my-cookie', {useLocalStorage: false});

Note that if the cookie is not existing for a key, then the $.cookie() function will return null.

Using Limitations

Let’s see how to use the maxNumberOfCookies and maxChunkSize options to limit the size and the number of cookies to be created.

With this example, the content will be saved in multiple cookies of 1000 bytes each up to 30 cookies:

var value = $.cookie('my-cookie', "the-content-of-my-cookie-is-10000-bytes-long...", {useLocalStorage: false, maxChunkSize  = 1000, maxNumberOfCookies = 30, expires: 365, path: "/" });

Limitations

Users have to be aware of the limitations of this enhanced plugin. Depending on the browser, the values of the maxChunkSize and the maxNumberOfCookies options should be different. In the worse case, some cookies (or cookies chunks) may simply not be created by the browser. As stated in the RFC 2109, the web applications have to take that fact into account, and be able to gracefully cope with this.

Future Enhancements

In the future, this extension should detect the browser where it runs, and setup the maxChunkSize and the maxNumberOfCookies parameters automatically depending on the cookies limitation of each browser.

Conclusion

I had to create this extension to the jQuery Cookie plugin to be able to store the resultsets returned by some web service endpoints. It is only used to limit the number of queries sent to these endpoints. Since the values returned by the endpoints are nearly static, that they are loaded at each page view and that they are a few kilobytes big, I had to find a way to save that information in the browser, and to overcome the size limitation of the cookies if possible. I also needed to be able to cope with older versions of browsers that only supports cookies. In the worse case scenario, the browser will simply send the request to the endpoints at each page load for the special use-cases where nothing works: not the cookies and not the localStorage. But at least, my application will benefit of this enhancement from the 95% of the users were one of these solutions works.

Open Semantic Framework Running on Micro Instances

After releasing the new Open Semantic Framework Installer, we started to test it on machines with all kind of different specifications: different CPU limits, different amount of memory, etc. One of the setup that caught our attention was Amazon’s EC2 Micro Instance.

The Micro Instance is a virtual server type that has been introduced by Amazon a little bit more than a year ago. As described by Amazon, Micro Instances are:

Instances of this family provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are well suited for lower throughput applications and web sites that consume significant compute cycles periodically.

We were intrigued by this particular type of instance because we wanted to know how the complete Open Semantic Framework stack could operate on such a small server instance.

Micro Instance Specifications

The Micro Instance’s specifications are as follow:

  • 613 MB memory
  • Up to 2 EC2 Compute Units (for short periodic bursts)
  • 32-bit or 64-bit platform
  • I/O Performance: Low

Note that a EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Installing The Stack

Installing the stack on the Amazon Micro Instance, using the OSF Installer, is not the fastest experience in the World. In fact, installing the complete stack takes up to 10 hours (5 minutes of your time, but compiling and installing everything takes about 10 hours of CPU time).

The problem is that installing OSF is a CPU intensive task, while the Micro instance is not. The micro instance can sustain small CPU bursts, but it can’t sustain the creation and compilation of the entire stack. That means that the CPU cycles won’t be available to the instance, and that the CPU consumption of that instance will be throttled by Amazon, which will significantly slow down the installation process.

However, as you will see below, once OSF is installed on the Micro instance, the complete stack responds perfectly to all queries sent to it.

Creating an AMI

The only time you have to spend 10 hours to install the OSF stack on an Amazon Micro Instance is the first time. After that, you would only have to create an Amazon AMI from that vanilla OSF instance for future use. If you proceed that way, you will lower the installation time from 10 hours to a few minutes.

Reading and Searching Data

The testing we did for reading and searching data from structWSF shows that performances are as good as the ones you would get from a small instance with a normal workload. The Crud: Read and the Search structWSF endpoints are fully responsive and operational.

Creating, Updating and Deleting Data

The testing we did for creating, updating and deleting entire datasets takes more time than with a small instance even if the instance is dedicated to that only task, without any other queries processed by the instance at the same time. The reason for this decrease in performances is due to the CPU throttling done by Amazon for this kind of more CPU intensive task. However, since individual records creation, updating and deletion creates “CPU Peaks”, such isolated create/update/delete queries doesn’t greatly affect the overall performances of the instance.

What This Type Of Instance Is Good For?

We found that such small instances were perfect for data collection activities performed by a single person, or a small group of collaborators. We also found that it could be used by low-traffic websites such as personal web portal, personal blogs, etc. The complete OSF stack is fully responsive and our analysis shows that the resources (CPU and Memory) are stable and responsive with a normal workload.

Conclusion

Such a small server instance can easily be used to create a personal data collection endpoint, or a personal, or small, data presentation portal such as Mike’s semantic web Sweet Tools. It is well suited for data portals that require reading and searching of data with occasional data changes (addition, removal and modification of instance records).

Volkswagen UK’s Search Engine Powered by structWSF

It is now official, Volkswagen UK‘s search engine is now powered by structWSF. Their new contextual search engine has been released last Friday. I covered the underlying architecture in one of my recent blog post: Volkswagen's RDF Data Management Workflow.

 

 

John Streit, head of technology at Tribal DDB, described the two key advantages of using the structWSF (part of the Open Semantic Framework (OSF)) for their website in an interview with Wired UK:

The first is that it gives you a single place to access data. Streit explains: “Applications often need to retrieve data from multiple sources which adds complexity and development time. By using this technology we can get everything we need from a single place which drastically lowers development time and running costs.” Furthermore the exposure of data improves search and means that it can be repurposed in new and imaginative ways.

The Open Semantic Framework Installer

We are excited to introduce the first Open Semantic Framework installation script. This new installer application will install and configure the entire Open Semantic Framework stack for you. It will take about 10 minutes of your time, and will process in the background for a few hours while everything necessary to build the OSF stack is downloaded and compiled. Open Semantic Framework Installer

The only thing you have to do to run the OSF Installer is to issue the few commands outlined below, and then to answer a few questions in the process (which, since most of them use the standard default values, is pretty easy).

The OSF Installer is a major addition to the Open Semantic Framework since it now enables a greater number of people (mere mortals) to install and use the stack, and it enables much faster deployment of the system.

The full installation manual, where each of the steps performed by the installer is explained in detail, is available as a reference here.

Requirements

The current version of the Open Semantic Framework Installer is fully operational on:

  1. Ubuntu 10.04 (Lucid)
  2. 32 Bits Operating System
  3. Access to internet from the server
  4. 5GIG of disk space on the partition where you are installing OSF

Eventually this installer will be upgraded for 64-bits operating systems, and for other Linux distributions. Also, the current installer should work on newer versions of Ubuntu, but it has only been tested to date on the latest LTS version.

Installing the Open Semantic Framework

The only manual steps need to do to install the Open Semantic Framework are to:

  1. Create a folder where to install OSF on your server
  2. Download the osf-install.zip installation package
  3. Make the osf-install.sh installation script executable
  4. Run the osf-install.sh installation script
  5. Answer the questions asked by the installer

Here are the commands you have to run:

1
2
3
4
5
6
cd /mnt/
sudo wget https://github.com/downloads/structureddynamics/Open-Semantic-Framework-Installer/osf-installer-v1.0a4.zip
sudo unzip osf-installer-v1.0a4.zip
cd `ls -d structureddynamics*/`
sudo chmod 755 osf-install.sh
./osf-install.sh

conStruct and structWSF Upgrades

In the process, both conStruct and structWSF have been enhanced to enable automatic upgrading in the future. Starting with structWSF version 1.0a92 and conStruct version 6.x-1.0-beta9, future upgrades should be done automatically using automatic upgrading procedures.

However, to enable this, existing users will have to upgrade their current versions manually to establish the new automatic upgrades baseline.

Next Steps

Once you have installed the OSF stack, you next query the structWSF Web service endpoints, and import datasets using conStruct. Here are a few things you can do to start exploring the Open Semantic Framework:

  1. Start exploring structWSF
  2. Start exploring conStruct
  3. Start exploring Ontologies usage in OSF
  4. Start importing and manipulating datasets
  5. Start exploring the Open Semantic Framework architecture
  6. Start playing with the structWSF web service endpoints

Since everything is installed on your server, so you only have to play with the stack now. If you break something, just ping us on the mailing list or re-install it without worrying about each installation steps!

Help

It may be possible that you experience some issues with this new OSF Installer. If that is the case, I would suggest your to make an outreach to the Open Semantic Web Mailing List so that we fix it on the Git repository.

Just write an email that includes the specifications of the server where you are trying to install OSF on. Then tell us where the issue happens in the installation process. Also add any logs that could be helpful in debugging the issue.

Conclusion

This is the first version of the OSF installer, but this is a real balm for installing OSF. As noted, this installer will eventually be upgraded to support 64-bit servers and other Linux distributions. Also, any help improving this installer from Bash wizards would naturally be greatly welcomed.

Role and Use of Ontologies in the Open Semantic Framework

Ontologies are to the Open Semantic Framework what humans were to the Mechanical Turk. The hidden human in the Mechanical Turk was orchestrating all and every chess move. However, to the observers, the automated chess machine was looking just like it: a new kind of intelligent machine. We were in 1770.

Ontologies plays exactly the same role for the Open Semantic Framework (OSF): they orchestrate all and every moves for all the pieces within OSF. They are what instructs structWSF, the Semantic Components, conStruct, and all other derivate pieces of user interfaces how to behave.

In this (lengthy) blog post, I will present the main ontologies that have an impact on different parts of OSF. We will see how different ontology classes and properties, and how the description of the records indexed in the system, can impact the behaviors of OSF.

In addition to this post, Mike has also published a blog post today that overviews the overall OSF ontology modularization and architecture.

Continue reading ‘Role and Use of Ontologies in the Open Semantic Framework’

Moving Projects from Google Code to GitHub

Last week we slowly migrated Structured Dynamics‘ Google Code Projects to GitHub.We have been thinking about moving to GitHub for some time now, but we only wanted to move projects to it if no prior history and commits were dropped in the process. One motivation for the possible change has been the seeming lack of support by Google for certain long-standing services: we are seeing disturbing trends across a number of existing services. We also needed a migration process that would work with all of our various projects, without losing a trunk, branch, tag or commits (and their related comments).

It was not until recently that I found a workable process. Other people have successfully migrated Google Code SVN projects to GitHub, but I had yet to find a consolidated guide to do it. It is for this last reason that I write this blog post: to help people, if they desire, to move projects from Google Code to GitHub.

Moving from Google Code to GitHub

The protocol outlined below may appear complex, but it looks more intimidating than it really is. Moving a project takes about two to five minutes once your GitHub account and your migration computer is properly configured.

You need four things to move a Google Code SVN project to GitHub:

  1. A Google Code project to move
  2. A GitHub user account
  3. SSH keys, and
  4. A migration computer that is configured to migrate the project from Google Code to GitHub. (in this tutorial, we will use a Ubuntu server; but any other Linux/Windows/Mac computer, properly configured, should do the job)

Create GitHub Account

If you don’t already own a GitHub account, the first step is to create one here.

Create & Configure SSH Keys

Once your account has been created, you have to create and setup the SSH keys that you will use to commit the code into the Git Repository on GitHub:

  1. Go to the SSH Keys Registration page of your account
  2. If you already have a key, then add it to this page, otherwise read this manual to learn how to generate one

Configure Migration Server

The next step is to configure the computer that will be used to migrate the project. For this tutorial, I use a Ubuntu server to do the migration, but any Windows, Linux or Mac computer should do the job if properly configured.

The first step is to install Git and Ruby on that computer:

1
 sudo apt-get install git-core git-svn ruby rubygems

To perform the migration of a Google Code SVN project to GitHub, we are using a Ruby application called svn2git that is now developed by Kevin Menard. The next step is to install svn2git on that computer:

1
 sudo gem install svn2git --source http://gemcutter.org

Migrate Project

Before migrating your project, you have to link the Google Code committers to GitHub accounts. This is done by populating a simple text file that will be given as input to svn2git.

Open the authors.txt file into a temporary folder:

1
 sudo vim /tmp/authors.txt

Then, for each author, you have to add the mapping between their Google Code and GitHub accounts. If a Google Code committer does not exist on GitHub, then you should map it to your own GitHub account.

1
2
 (no author) = Frederick Giasson <fred@f...com>
 fred@f...com = Frederick Giasson <fred@f...com>

The format of this authors.txt file is:

1
 Google-Account-Username = Name-Of-Author-On-GitHub <Email-Of-Author-On-Github

Take note of the first Google Code committer (no author) mapping. This link is required for every authors.txt file. This placeholder is used to map the initial commit performed by the Google Code system. (When Google Code initializes a new project, it uses that username for creating the first commit of any project.)

When you are done, save the file.

Now that set up is complete, you are ready to migrate your project. First, let’s create the folder that will be used to checkout the SVN project on the server, and then to push it on GitHub.

1
2
3
cd /tmp/
mkdir myproject
cd myproject

In this tutorial, we have a normal migration scenario. However, your migration scenario may differ from this one. It is why I would suggest you check out the different scenarios that are supported by svn2git document. Change the following command accordingly. Let’s migrate the Google Code SVN Project into the local Git repository:

1
 /var/lib/gems/1.8/bin/svn2git http://myproject.googlecode.com/svn --authors /tmp/authors.txt --verbose

Make sure that no errors have been reported during the process. If it is the case, then refer to the Possible Errors and Fixes section below to troubleshoot your issue.

The next step is to create a new GitHub repository where to migrate the SVN project. Go to this GitHub page to create your new repository. Then you have to configure Git to add a remote link, from the local Git repository you created on your migration computer, to this remote GitHub repository:

1
 git remote add origin git@github.com:you-github-username/myproject.git

Finally, let’s push the local Git repository master, branches and tags to GitHub. The first thing to push onto GitHub is the SVN’s trunk. It is done by running that command:

1
 git push -u origin master

Then, if your project has multiple branches and tags, you can push them, one by one, using the same command. However, you will have to replace master by the name of that branch or tag. If you don’t know what is the exact name of these branches or tags, you can easily list all of them using this Git command:

1
 git show-ref

Once you have progressed through all branched and tags, you are done. If you take a look at your GitHub project’s page, you should see that the trunk, branches, tags and commits are now properly imported into that project.

Possible Errors And Fixes

Fatal Error: Not a valid object name

There are a few things that can go wrong while trying to migrate your project(s).

One of the errors I experienced is a "fatal" error message "Not a valid object name". To fix this, we have to fix a line of code in svn2git. Open the migration.rb file. Check around the line 227 for the method fix_branches(). Remove the first line of that method, and replace the second one by:

1
 svn_branches = @remote.find_all { |b| !@tags.include?(b) && b.strip =~ %r{^svn\/} }

Error: author not existing

While running the svn2git application, the process may finish prematurely. If you check the output, you may see that it can’t find the match for an author. What you will have to do is to add that author to your authors file and re-run svn2git. Otherwise you won’t be able to fully migrate the project.

I’m not quite sure why these minor glitches occurred during my initial migrate, but with the simple fixes above you should be good to go.




This blog is a regularly updated collection of my thoughts, tips, tricks and ideas about my semantic Web researches and related software development.


RSS
454


Follow

Get every new post on this blog delivered to your Inbox.

Join 18 other followers:

Or subscribe to the RSS feed by clicking on the counter:




RSS
454