NOW Portal Introduction from City of Winnipeg

Introduction

I am proud to announce the new NOW (Neighbourhoods Of Winnipeg) semantic web portal! This new and innovative semantic web portal was publicly announced by the Mayor of Winnipeg City last week.

The NOW (Neighbourhoods of Winnipeg) portal is “a new Web portal (the “Portal”) produced by the City of Winnipeg to provide broad, dynamic and interactive access to local and neighbourhood information. Designed for easy access and use by all citizens, businesses, community organizations and Governments, the information on the site includes municipal data, census and demographic information, economic development information, historical data, much spatial and mapping information, and facilities for including and sharing data by external groups and constituencies.”

I would suggest you to read Mike Bergman’s blog post about this new semantic web portal to have the proper background about that initiative by the city of Winnipeg and how it uses the OSF (Open Semantic Framework) as its foundational technology stack.

This project has been the springboard that led to the Open Semantic Framework version 1.1. Multiple pieces of the framework have been developed in relation to this project, and more particularly pieces like the sWebMap semantic component and several improvements to the structWSF web services endpoints and conStruct modules for Drupal 6.

Development of the Portal

The development plan of this portal is composed of four major areas:

1. Development of the data structure of the municipal domain by creating a series of ontologies
2. Conversion of existing data asset using this new data structure
3. Creation of the web portal by creating its design and by developing all the display templates
4. Creation of new tools to let users interact with the data available on the portal

Structured Dynamics has been involved in #1, #2 and #4 by providing design and development resources, technology transfer sessions and material and supporting internal teams to create, maintain and deploy their 57 publicly available datasets.

The Data Structure

This technology stack does not have any meaning without the proper data and data structures (ontologies) in place. This gold mine of information is what drives the functionality of the portal.

The portal is driven by 12 ontologies: 2 internal and 10 external. The content of the 57 publicly available datasets is defined by the classes and properties defined in one of these ontologies.

The two internal ontologies have been created jointly by Structured Dynamics and the City of Winnipeg, but they are extended and maintained by the city only.

These ontologies are maintained using two different kind of tools:

1. Protege
2. structOntology

Protege is used for the big development tasks such as creating a big number of classes and properties, to do a big reorganization of the classes structure, etc.

structOntology is used for quick ontological changes to have an immediate impact on the behaviors of the portals such as label changes, SCO ontology property assignments to change the behavior of some of the tools that exist in the portal, etc.

structOntology can also be used by portal users to understand the underlying data structure used to define the data available on the portal. All users have access to the reading mode of the tool which let them browse, search and export the loaded ontologies on the portal.

The Data

Except for rare exceptions such as the historical photos, no new data has been created by the City of Winnipeg to populate this NOW portal. Most of its content comes from existing internal sources of data such as:

• Conventional relational databases
• GIS (Geographic Information System) on-top of relational databases
• Spreadsheets

All of the conventional relation databases and legacy data from the GIS systems has been converted into RDF using the FME Workbench ETL system. All of the FME workbench templates are mapping the relational data into RDF using the ontologies loaded into the portal. All of the geolocated records that exist in the portal come from this ETL process and have been converted using FME.

Some smaller datasets come from internal spreadsheets that got modified to comply with the commON spreadsheet format that is used to convert spreadsheet (CSV/TSV) data files into RDF.

All of the dataset creation and maintenance is managed internally by the City of Winnipeg using one of these two data conversion and importation processes.

Here are some internal statistics of the content that is currently accessible on the NOW portal.

General Portal

These are statistics related to different functionalities of the portal.

• Number of neighbourhoods: 236
• Number of community areas: 14
• Number of wards: 15
• Number of neighbourhood clusters: 23
• Number of major site sections: 7
• Total number of site pages: 428,019
  • Static pages: 2,245
  • Record-oriented pages: 425,874
  • Dynamic (search-based) pages: infinite
• Number of documents: 1,017
• Number of images: 2,683
• Number of search facets: 1,392
• Number of display templates: 54
• Number of links: 1,067
  • External links: 784
  • Internal links: 283

Site Data

These statistics show the things that are available via the portal, what are their types, their properties, what is the quantity of data that is searchable, manipulable and exportable from the portal.

• Number of datasets: 57
• Number of records: 425,874
  • Number of geolocational records: 418,869
    • Point of interest (POI) records: 193,272
    • Polygon records: 218,602
    • Path (route) records: 6,995
• Number of classes (types): 84
• Number of properties: 1,308
• Number of triple assertions: 8,683,103

Sharing Content

An important aspect of this portal is that all of the content is contextually available, in different formats, to all of the users of the portal. Whether you are browsing content within datasets, searching for specific pieces of content, or looking at a specific record page, you always have the possibility to get your hands on the content that is being displayed to you, the user, with a choice of five different data formats:

• structXML
• structJSON
• RDF+XML
• N3/Turtle
• irJSON

Export Page Content

All content pages can be exported in one of the formats outlined above. In the bottom right corner of these pages you will see a Export button that you can click to get the content of that page in one of these formats.

Export Search Content

Every time you do a search on the portal, you can export the results of that search in one of the formats outlined above. You can do that by selecting the Export tab, and by selecting one of the formats you want to use for exporting the data.

Export Datasets

You can export any publicly available dataset from the portal. These datasets have to be exported in slices if they are too big to fit in a single slice. The datasets can be exported in one of the formats mentioned above.

Export Census

Users also have the possibility to export census data, from the census section of the portal, in spreadsheets. They only have to select the Tables tab, and then to click the Export Spreadsheet button.

Export Ontologies

The export functionality would not be complete without the ability to consult and export the ontologies that are used to describe the content exposed by the portal. These ontologies can be read from the ontologies reader user interface, or can be exported from the portal to be read by external ontologies management tools such as Protege.

Portal Design

The portal is using Drupal 6 as its CMS (Content Management System). The Drupal 6 instance communicates with structWSF using the conStruct module, which acts as a bridge between a Druapal portal and a structWSF web service network.

Here are the main design phases that have been required to create the portal:

1. Creation of the portal’s design, and the Drupal 6 theme that implements it
2. Creation of the Search and Browse results templates
3. Creation of the individual records’ page design and templates based on their type
4. Creation of the sWebMap search results templates.

The portal’s design has been created internally by the City of Winnipeg and by Tactica based on the Citizen DAN demo. Tactica also worked on another Citizen DAN like portal called MyPeg.ca.

Semantic Components

The NOW Web portal is using a series of tools that are called the Semantic Components. These are a set of Flash and JavaScript tools that can be embedded within any web page and that can easily communicate with structWSF instance(s). They display information in all kinds of charts, they can display document reading widgets, they can create dashboards of structured data, etc. The initial set of Semantic Components was developed for the MyPeg.ca project back in November 2010. This was before Steve Jobs announced that Apple would not support Adobe Flash, and far before Google announced that it would drop support for it as well.

Since the NOW portal wanted to re-use as much as possible to lower the development cost related to the portal, they choose to use the complete OSF stack which includes these Semantic Components.

However, when we participated in developing this new NOW portal, we did extended the set of Semantic Components by creating the most complex Semantic Component: the sWebMap. However, because of the two announcements mentioned above, we choose to move forward and to create the sWebMap Semantic Component using JavaScript instead of Flash. The other Semantic Component tools that have been developed in Flash have not yet been ported into JavaScript.

Conclusion

The new NOW semantic web portal’s main asset is its data: how it can be searched (with traditional search engines or using a semantic component to search, browse, filter and localize results), displayed and exported. This portal has been developed using a completely free and open source semantic platform that has been developed from previous projects that open sourced their code.

I consider this portal a pioneer in the way municipal organization will provide new online services to their citizens and to the commercial enterprises based on the quality of the data that will be exposed via such Web portals.

I am please to announce the release of the new sWebMap Semantic Component in JavaScript. This new mapping component is a standalone JavaScript application that can be integrated on any new or existing web sites and that interact with an Open Semantic Framework (OSF) instance to search, browse, filter and display with geographically-located information on an interactive map.

Features

The sWebMap is a rich mapping tool that can easily be integrated on any webpage, and that can be extensively customized. The sWebMap does support these features:

• Full text search for searching and displaying results on a map
• Extensive filtering capabilities
• Filtering by dataset source
• Filtering by type
• Filtering by attribute/value
• Filtering of records that belongs to a specific geographic region
• Display of record on the map using:
• Different markers depending on the type of record to display (determined by the ontologies)
• Polygon shapes for records that refers to a geographic region
• Polyline shapes for records that refers to a geographically-located path
• Templating of records in a resultset depending on their type
• Templating of records’ preview, displayed in an overlay window, depending on their type
• Persist records on the map accros searches and filtering operations
• Supports map sessions
• Save map sessions
• Load saved map sessions
• Delete saved map sessions
• Share saved map sessions
• Supports a multiple-maps mode
• Three focus maps are available under the main map
• Each map focus on a particular region of the main map
• User can switch between focus map to see different records in different region

Normal Mode

Here is what the default sWebMap, in normal mode, using a few datasets related to the city of Iowa looks like. You can also interact with this sWebMap instance directly on the Citizen DAN demo website here.

Multiple Windows Mode

Here is what the default sWebMap, in multiple windows mode, using a few datasets related to the city of Iowa looks like. You can also interact with this sWebMap instance directly on the Citizen DAN demo website here.

Under the Hood: The Open Semantic Framework

Each sWebMap component communicates with an OSF (Open Semantic Framework) instance. More specifically, a sWebMap component will send Search/Filtering queries to a geo-enabled structWSF Search web service endpoint.

Depending on the options you had specified when you created the sWebMap control, each time you move (option), zoom (option) or change the filtering criterias, this will send a query to the Search endpoint. The sWebMap control then requests JSON formatted resultset and display the results to the user.

This means that to implement the sWebMap component on your website, you will need to have:

• a running structWSF instance
• some datasets that have geo-localized information
• properly defined records and ontologies classes

Download

You can immediately download the entire code source from this GitHub reposiroty:

Installation

Installing the sWebMap component is really easy. In fact, you only have to load a few JavaScript and CSS files, to defined a <div></div> container for the map, and to create a sWebMap component object, which is a single line of code.

Additionally, you can initialize the sWebMap component with one of the multiple options available.

Refer you to the Usage section of the sWebMap component to know exactly how to install and setup a sWebMap component instance.

Resources

Here are some additional resources related to the sWebMap component:

• sWebMap complete documentation
• sWebMap user manual
• OSF Installer (install the Open Semantic Framework to feed your sWebMap component)
• sWebMap Demo (normal mode)
• sWebMap Demo (multiple windows mode)

In any case, if one of these limits is reached, the cookie is simply not created the browser. This is fine, because web developer expects cookies to fail from time to time, and the system they develop has to cope with this unreliableness. However, this situation can sometimes become frustrating, and it is why I wanted to extend the default behavior of the jQuery Cookie plugin with a few more capabilities.

This extension to the jQuery Cookie plugin adds the capability to save content that is bigger than 4096 bytes long using two different mechanism: the usage of HTML5′s localStorage, or the usage of a series of cookies where the content is chunked and saved. This extension is backward compatible with the jQuery Cookie plugin and its usage should be transparent to the users. Even if existing cookies have been created with the normal Cookie plugin, they will still be usable by this new extension. The usage syntax is the same, but 3 new options have been created.

Now, let’s see how this plugin works, how developers should use it, what are its limitations, etc.

You can immediately download the jQuery Extended Cookie plugin from here:

Limitations Of Cookies

First, let’s see what the RFC 2109 says about the limitations of cookies in web browsers. Browsers should normally have these implementation limits (see section 6.3):

   Practical user agent implementations have limits on the number and
   size of cookies that they can store.  In general, user agents' cookie
   support should have no fixed limits.  They should strive to store as
   many frequently-used cookies as possible.  Furthermore, general-use
   user agents should provide each of the following minimum capabilities
   individually, although not necessarily simultaneously:

      * at least 300 cookies
      * at least 4096 bytes per cookie (as measured by the size of the
        characters that comprise the cookie non-terminal in the syntax
        description of the Set-Cookie header)
      * at least 20 cookies per unique host or domain name

   User agents created for specific purposes or for limited-capacity
   devices should provide at least 20 cookies of 4096 bytes, to ensure
   that the user can interact with a session-based origin server.

   The information in a Set-Cookie response header must be retained in
   its entirety.  If for some reason there is inadequate space to store
   the cookie, it must be discarded, not truncated.

   Applications should use as few and as small cookies as possible, and
   they should cope gracefully with the loss of a cookie.

New Options

Before I explains how this extension works, let me introduce three new options that have been added to the Cookie plugin. These new options will be put into context, and properly defined later in this blog post.

• maxChunkSize - This defines the maximum number of bytes that can be saved in a single cookie. (default: 3000)
• maxNumberOfCookies - This is the maximum number of cookies that can be created for a single domain name. (default: 20)
• useLocalStorage - This tells the extended Cookie plugin to use the HTML5′s localStorage capabilities of the browser instead of a cookie to save that value. (default: true)

How Does This Extension Works?

As I said in the introduction of this blog post, this extension to the jQuery Cookie plugin does two things:

1. It uses the HTML5 localStorage capabilities of the browser if this feature is available instead of relying on the cookies. However, if cookies are needed by the developer, this feature can be turned off with the useLocalStorage = false option
2. If the localStorage option is disable, or simply not available on a browser, and if the content is bigger than the limit of the size of a cookie, then this extension will chunk the input content, and save it in multiple cookies

If the useLocalStorage is true, then the plugin will try to see if the HTML5 localStorage mechanism is available on the browser. If it is, then it will use that local storage to save and retrieve content to the browser. If it is not, then the plugin will act like if useLocalStorage is false and the process will continue by using cookies to save and read that content from the browser.

If useLocalStorage is false, or if the HTML5 localStorage mechanism is not available on the browser, then the plugin will check if the content is bigger than the maxChunkSize option, than all the chunks will be saved in different cookies until it reaches the limit imposed by the maxNumberOfCookies option.

If cookies are used, then two use-cases can happen:

1. The content is smaller than or equal to maxChunkSize
2. The content is bigger than maxChunkSize

If the content is smaller than or equal to maxChunkSize than only one cookie will be created by the browser. The name of the cookie will be the value provided to the key parameter.

If the content is bigger than maxChunkSize than multiple cookies will be created, one per chunk. The convention is that the name of the first cookie is the value provided to the key parameter. The name of the other chunks is the value provided to the key parameter with the chunk indicator ---ChunkNum append to it. For example, if we have a cookie with a content of 10000 bytes that has maxChunkSize defined to 4000 bytes, then these three cookies would be created:

• cookie-name
• cookie-name---1
• cookie-name---2

Usage

Now, let’s see how this extended jQuery Cookie plugin should be used in your code. The usage of the extension is no different from the usage of the normal jQuery Cookie plugin. However, I am showing how to use the new options along with how to use the plugin in general.

Create a Cookie

Let’s create a cookie that expires in 365 days and where the path is the root:

$.cookie('my-cookie', "the-content-of-my-cookie", { expires: 365, path: "/" });

By default, this value will be persisted in the localStorage if the browser supports it, and not in a cookie. So, let’s see how to force the plugin to save the content in a cookie by using the useLocalStorage option:

$.cookie('my-cookie', "the-content-of-my-cookie", {useLocalStorage: false, expires: 365, path: "/" });

Delete a Cookie

Let’s see how a cookie can be deleted. The method is simply to put null as the value of the cookie. This will instruct the plugin to remove the cookie.

$.cookie('my-cookie', null);

With that call, the plugin will try to remove my-cookie both in the localStorage and in the cookies.

Read a Cookie

Let’s see how we can read the content of a cookie:

var value = $.cookie('my-cookie');

With this call, value will get the content that has been saved in the localStorage, or the cookies. This will depend if the localStorage was available in the browser.

Now, let’s see how to force reading the cookies by bypassing the localStorage mechanism:

var value = $.cookie('my-cookie', {useLocalStorage: false});

Note that if the cookie is not existing for a key, then the $.cookie() function will return null.

Using Limitations

Let’s see how to use the maxNumberOfCookies and maxChunkSize options to limit the size and the number of cookies to be created.

With this example, the content will be saved in multiple cookies of 1000 bytes each up to 30 cookies:

var value = $.cookie('my-cookie', "the-content-of-my-cookie-is-10000-bytes-long...", {useLocalStorage: false, maxChunkSize  = 1000, maxNumberOfCookies = 30, expires: 365, path: "/" });

Limitations

Users have to be aware of the limitations of this enhanced plugin. Depending on the browser, the values of the maxChunkSize and the maxNumberOfCookies options should be different. In the worse case, some cookies (or cookies chunks) may simply not be created by the browser. As stated in the RFC 2109, the web applications have to take that fact into account, and be able to gracefully cope with this.

Future Enhancements

In the future, this extension should detect the browser where it runs, and setup the maxChunkSize and the maxNumberOfCookies parameters automatically depending on the cookies limitation of each browser.

Conclusion

I had to create this extension to the jQuery Cookie plugin to be able to store the resultsets returned by some web service endpoints. It is only used to limit the number of queries sent to these endpoints. Since the values returned by the endpoints are nearly static, that they are loaded at each page view and that they are a few kilobytes big, I had to find a way to save that information in the browser, and to overcome the size limitation of the cookies if possible. I also needed to be able to cope with older versions of browsers that only supports cookies. In the worse case scenario, the browser will simply send the request to the endpoints at each page load for the special use-cases where nothing works: not the cookies and not the localStorage. But at least, my application will benefit of this enhancement from the 95% of the users were one of these solutions works.

The Micro Instance is a virtual server type that has been introduced by Amazon a little bit more than a year ago. As described by Amazon, Micro Instances are:

Instances of this family provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are well suited for lower throughput applications and web sites that consume significant compute cycles periodically.

We were intrigued by this particular type of instance because we wanted to know how the complete Open Semantic Framework stack could operate on such a small server instance.

Micro Instance Specifications

The Micro Instance’s specifications are as follow:

• 613 MB memory
• Up to 2 EC2 Compute Units (for short periodic bursts)
• 32-bit or 64-bit platform
• I/O Performance: Low

Note that a EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Installing The Stack

Installing the stack on the Amazon Micro Instance, using the OSF Installer, is not the fastest experience in the World. In fact, installing the complete stack takes up to 10 hours (5 minutes of your time, but compiling and installing everything takes about 10 hours of CPU time).

The problem is that installing OSF is a CPU intensive task, while the Micro instance is not. The micro instance can sustain small CPU bursts, but it can’t sustain the creation and compilation of the entire stack. That means that the CPU cycles won’t be available to the instance, and that the CPU consumption of that instance will be throttled by Amazon, which will significantly slow down the installation process.

However, as you will see below, once OSF is installed on the Micro instance, the complete stack responds perfectly to all queries sent to it.

Creating an AMI

The only time you have to spend 10 hours to install the OSF stack on an Amazon Micro Instance is the first time. After that, you would only have to create an Amazon AMI from that vanilla OSF instance for future use. If you proceed that way, you will lower the installation time from 10 hours to a few minutes.

Reading and Searching Data

The testing we did for reading and searching data from structWSF shows that performances are as good as the ones you would get from a small instance with a normal workload. The Crud: Read and the Search structWSF endpoints are fully responsive and operational.

Creating, Updating and Deleting Data

The testing we did for creating, updating and deleting entire datasets takes more time than with a small instance even if the instance is dedicated to that only task, without any other queries processed by the instance at the same time. The reason for this decrease in performances is due to the CPU throttling done by Amazon for this kind of more CPU intensive task. However, since individual records creation, updating and deletion creates “CPU Peaks”, such isolated create/update/delete queries doesn’t greatly affect the overall performances of the instance.

What This Type Of Instance Is Good For?

We found that such small instances were perfect for data collection activities performed by a single person, or a small group of collaborators. We also found that it could be used by low-traffic websites such as personal web portal, personal blogs, etc. The complete OSF stack is fully responsive and our analysis shows that the resources (CPU and Memory) are stable and responsive with a normal workload.

Conclusion

Such a small server instance can easily be used to create a personal data collection endpoint, or a personal, or small, data presentation portal such as Mike’s semantic web Sweet Tools. It is well suited for data portals that require reading and searching of data with occasional data changes (addition, removal and modification of instance records).

John Streit, head of technology at Tribal DDB, described the two key advantages of using the structWSF (part of the Open Semantic Framework (OSF)) for their website in an interview with Wired UK:

The first is that it gives you a single place to access data. Streit explains: “Applications often need to retrieve data from multiple sources which adds complexity and development time. By using this technology we can get everything we need from a single place which drastically lowers development time and running costs.” Furthermore the exposure of data improves search and means that it can be repurposed in new and imaginative ways.

The only thing you have to do to run the OSF Installer is to issue the few commands outlined below, and then to answer a few questions in the process (which, since most of them use the standard default values, is pretty easy).

The OSF Installer is a major addition to the Open Semantic Framework since it now enables a greater number of people (mere mortals) to install and use the stack, and it enables much faster deployment of the system.

The full installation manual, where each of the steps performed by the installer is explained in detail, is available as a reference here.

Requirements

The current version of the Open Semantic Framework Installer is fully operational on:

1. Ubuntu 10.04 (Lucid)
2. 32 Bits Operating System
3. Access to internet from the server
4. 5GIG of disk space on the partition where you are installing OSF

Eventually this installer will be upgraded for 64-bits operating systems, and for other Linux distributions. Also, the current installer should work on newer versions of Ubuntu, but it has only been tested to date on the latest LTS version.

Installing the Open Semantic Framework

The only manual steps need to do to install the Open Semantic Framework are to:

1. Create a folder where to install OSF on your server
2. Download the osf-install.zip installation package
3. Make the osf-install.sh installation script executable
4. Run the osf-install.sh installation script
5. Answer the questions asked by the installer

Here are the commands you have to run:

1 2 3 4 5 6

cd /mnt/ sudo wget https://github.com/downloads/structureddynamics/Open-Semantic-Framework-Installer/osf-installer-v1.0a4.zip sudo unzip osf-installer-v1.0a4.zip cd `ls -d structureddynamics*/` sudo chmod 755 osf-install.sh ./osf-install.sh

conStruct and structWSF Upgrades

In the process, both conStruct and structWSF have been enhanced to enable automatic upgrading in the future. Starting with structWSF version 1.0a92 and conStruct version 6.x-1.0-beta9, future upgrades should be done automatically using automatic upgrading procedures.

However, to enable this, existing users will have to upgrade their current versions manually to establish the new automatic upgrades baseline.

Next Steps

Once you have installed the OSF stack, you next query the structWSF Web service endpoints, and import datasets using conStruct. Here are a few things you can do to start exploring the Open Semantic Framework:

1. Start exploring structWSF
2. Start exploring conStruct
3. Start exploring Ontologies usage in OSF
4. Start importing and manipulating datasets
5. Start exploring the Open Semantic Framework architecture
6. Start playing with the structWSF web service endpoints

Since everything is installed on your server, so you only have to play with the stack now. If you break something, just ping us on the mailing list or re-install it without worrying about each installation steps!

Help

It may be possible that you experience some issues with this new OSF Installer. If that is the case, I would suggest your to make an outreach to the Open Semantic Web Mailing List so that we fix it on the Git repository.

Just write an email that includes the specifications of the server where you are trying to install OSF on. Then tell us where the issue happens in the installation process. Also add any logs that could be helpful in debugging the issue.

Conclusion

This is the first version of the OSF installer, but this is a real balm for installing OSF. As noted, this installer will eventually be upgraded to support 64-bit servers and other Linux distributions. Also, any help improving this installer from Bash wizards would naturally be greatly welcomed.

Ontologies plays exactly the same role for the Open Semantic Framework (OSF): they orchestrate all and every moves for all the pieces within OSF. They are what instructs structWSF, the Semantic Components, conStruct, and all other derivate pieces of user interfaces how to behave.

In this (lengthy) blog post, I will present the main ontologies that have an impact on different parts of OSF. We will see how different ontology classes and properties, and how the description of the records indexed in the system, can impact the behaviors of OSF.

In addition to this post, Mike has also published a blog post today that overviews the overall OSF ontology modularization and architecture.

Constituent Ontologies

Let’s take a look at the core ontologies used by the Open Semantic Framework. All these ontologies have been developed in relation to OSF. These, and other external ontologies, have the same role in OSF as the human does in the Mechanical Turk: they instruct the system how to behave.

Here is the list of the core ontologies:

1. The SCO Ontology (Semantic Component Ontology)
2. The WSF Ontology (Web Service Framework Ontology)
3. The AGGR Ontology (Aggregation Ontology)
4. The irON Ontology (Instance Record an Object Notation Ontology)
5. One or more domain ontologies, to capture the concepts and relationships for the purposes of a given OSF installation, and
6. Possibly UMBEL or other upper-level concept ontologies, used for linkages to external systems.

(Note: the internal wiki links to each of these ontologies also provides links to the actual ontology specifications on Github.)

A useful discussion of these ontologies and their interactions in an OSF instance is provided by the ontology modularization document. This current document focuses primarily on the specific properties and roles associated with them in an OSF installation.

Depending on the specific OSF installation, of course, multiple external ontologies may also be employed. Some of the common external ones used in an OSF installation are described by the external ontologies document. These external ontologies are important — indeed essential in order to ensure linkage to the external world — but have little to do with internal OSF control structures. That is why the rest of this discussion focused on internal ontologies only.

Summary Ontology Roles

Ontologies play pivotal roles across all parts of the framework. In a broad sense, the internal OSF ontologies are used for annotations, guiding interactions or relating concepts and information to other information. In specific terms, OSF ontologies may play one or more of these dozen or so roles:

1. Define record descriptions
2. Inform interface displays
3. Integrate different data sources
4. Define component selections
5. Define component behaviors
6. Guide template selection
7. Provide reasoning and inference
8. Guide content filtering (with and without inference)
9. Tag concepts in text documents
10. Help organize and navigate Web portals
11. Manage datasets and ontologies, and
12. Set access permissions and registrations.

In the remainder of this post, for each of these roles, we will see how ontologies affect numerous different parts of the OSF framework. These sections are presented in the order above.

Define Records Descriptions

A central role of ontologies in the Open Semantic Framework is their use to describe any kind of record that gets indexed and managed by the system. Since the framework indexes everything into the RDF data model, ontologies are needed as a schema to describe these RDF resources.

The irON ontology is specifically designed for record descriptions and notations. It interacts with all of the domain and (if used, UMBEL) upper level ontologies.

Inform Interface Displays

Ontologies have an impact in most of the user interfaces that display record information. The property that has the most impact is iron:prefLabel, which is used to display the label within the user interface that refers to a record or record attributes (properties). This label can be used within text, in a list control, in a tree control, or in any other kind of control that displays references to records.

Note: there are also other properties that are considered as fallbacks to iron:prefLabel if a record has no triples using the iron:prefLabel property. These include rdfs:label, dcterms:title, foaf:name, etc.

General User Interface Labels and Descriptions

There are a few properties that have an impact on most of the components of the OSF stack, most of which come from the irON ontology. Here is the list of these irON properties that impact other parts of the system, mainly related to different user interfaces:

User Interface ‘Short’ Labels

There are a few properties that impact most of the components of the OSF stack. Here is the list of SCO properties that impact other parts of the system, mainly related to different user interfaces:

Hierarchical Displays

The way ontologies define a class or a property structure also has an impact on different kinds of hierarchical displays. An example of this is the “Filter by Kinds” section of the structSearch and structBrowse modules. The possible filters that may be applied to a search query will be displayed to the user according to the hierarchy as defined in the ontologies.

Integrate Heterogeneous Data Sources

The principle reason why the Open Semantic Framework uses RDF and ontologies to describe all the data it indexes and manages is to facilitate data integration from multiple and heterogeneous data sources. The premise of using RDF and ontologies is:

This foundation leads to the extreme flexibility of the Open Semantic Framework. The rationale behind this flexibility, and its benefits, has been described in many locations within this wiki. You may also want to see this article on One of the Semantic Web's Core Added Value.

Ontologies have a dramatic — and positive — impact on the data integration and presentation tasks within an OSF instance.

Define Component Selections

A key aspect of the SCO Ontology is its use as the means to define what semantic components (or widgets) display what types of information within data records.

These assignments are done via the sControl component. The properties for this component define what components may display what type (class) of data records. Here is the list of SCO properties that impact the sControl’s behaviors:

Define Component Behaviors

In the Open Semantic Framework, one of the most important roles of ontologies is to define the interaction between different pieces of the system. Because of the extent of these interactions, this section is the longest and most detailed amongst all of the dozen or so ontology roles.

The SCO ontology can have multiple effects on multiple parts of an OSF instance. This section describes those interactions.

sMap Component

The sMap component had different behaviors depending on how its input record is described. Here is the list of SCO properties that will have an impact on the sMap’s behaviors:

sWebMap Component

The sWebMap component has different behaviors depending on how its input record is described. Here is the list of SCO and WGS84 properties that impact the behavior of an sWebMap:

sStory Component

The sStory component has different behaviors depending on how its input record is described. Here is the list of SCO properties that impact an sStory component:

sBarChart and sLinearChart Components

The sBarChart and the sLinearChart components exhibit different behaviors depending on how the input records that are enabled for these component types are described. Here is the list of the SCO properties that impact this behavior:

sRelationBrowser

The sRelationBrowser component exhibits different behaviors depending on how its input record is described. Here is the list of SCO properties that impact the sRelationBrowser component:

sDashboard

The sDashboard component exhibits different behaviors depending on how its input record is described. Here is the list of SCO properties that impact the sDashboard component:

Guide Visualization Template Selection

One of the core features of the conStruct set of Drupal modules is the ability to use different display templates depending on the types of records available. The selection of these templates is based on the types of those records and the type hierarchies described by the OSF ontologies. This section describes how these ontologies guide template selections.

As a refresher on templates and their use, see the Building conStruct Templates document. It describes how the templating engine works and how to create various templates.

Template Selection

Template selection is the action of binding an instance record to a display template based on its type. Three things are required to make this happen:

1. Instance records have to be typed
2. An ontological structure of type relationships (via subClasses) has to exist in one or more OSF ontology(ies), and
3. A template has to exist for the type of the instance record.

(Note: a specific template by type is not strictly required, since lacking a specific template for the target type, the system will invoke the nearest template up the parental chain in the governing ontology structure, eventually getting to the most generic template available, that for “thing”.)

Impact of Ontologies on Template Selection

conStruct’s templating engine selects record display templates based on the class hierarchy loaded on a OSF instance. It also uses inference on types to select the proper template for a given record.

Let’s say that we try to display information about a foaf:Person instance record. What the system attempts to do is to find a template that displays information about this kind of instance record. First, the foaf:Person type (class) has to be defined in the ontological structure of the OSF instance; if it is not, then no specific template will be selected and the system will default to using the owl_thing.html template (see below). If the type (class) is found, the system will next check to see if a template exists for that specific type. If one exists one, it will use the matching template. If one does not, it will next select the parent class of the type and try the match again. If it again fails, it will continue its test up the parental chain. If all tests fail, it will use the default owl_thing.html template. Whichever template is selected then becomes the basis for formatting and presenting the visual record display.

We can use a simple class hierarchy, matched to a simple set of available template files, to illustrate how ontologies impact the conStruct templating system.

owl:Thing
   |
   |
    --> foaf:Agent
            |
            |
            |--> foaf:Person
            |
            |
             --> foaf:Organization

  owl_thing.tpl
  foaf_agent.tpl
  foaf_organization.tpl

Now, let’s say that our OSF portal is about to display information about a foaf:Person record. As we can notice, there is no foaf_person.tpl template available for a foaf:Person. However, because of the ontology structure, the system next attempts to select a template from a parent class of that foaf:Person.

What the system would do is to check if there is a template available for a record of type foaf:Person. Since there is none, it would try to find one for a parent type, so in this case the foaf:Agent class. In our example, there is now a match. The templating engine thus uses the foaf_agent.tpl template to display information about the foaf:Person record.

Were the foaf_agent.tpl not to exist, then the templating engine would fall back to the owl_thing.tpl template, which is considered to be the “generic record display template”, or the template of last resort.

This design means that if:

1. the ontological structure changes over time, or
2. new templates get added to the system

then there may be an impact on how the record gets displayed.

The major advantage of this design is that more and more specific formatting templates may be added to an OSF installation over time, both improving the tailored look of results displays and accommodating more structure and relationships as they evolve.

Provide Reasoning and Inference

A standard use of ontologies is for reasoning and inference, and those used by OSF are no exception.

By extension, however, we can also use these same capabilities to check on data consistency and coherence. This is an important feature of the system since the system can detect if there are logical inconsistencies or logical incoherencies that have been developed by the system administrator during ontology growth and development. Having coherent and consistent ontologies means that we have the proper foundations to create consistent and coherent datasets of instance records.

See further the discussion on reasoning using Protégé.

Guide Content Filtering

Filtering data is the action of getting a subset of records from a complete dataset based on some selection criteria. In OSF, the predominant share of filtering is done using the structWSF Search Web service endpoint. The a minority of filtering is done using the SPARQL endpoint. It is also possible to filter via the AGGR aggregation ontology.

Possible filtering criteria for the Search endpoint are:

1. Filtering by type(s)
2. Filtering by attribute(s)
3. Filtering by attribute(s)/value(s)
4. Filtering by geo-localization (within a given geographical area)

These filtering activities are performed by different tools of the stack, such as:

• structSearch
• structBrowse
• sWebMap

These tools are impacted by the definition of the loaded ontologies. The filtering of the values by types, attributes and attributes/values requires an ontology class or an ontology property as filtering criteria.

Filtering with Inference

Also, the any Search query can be performed with inference enabled. Just like with the template selection section noted above, inference can have a big impact on the number and nature of returned results. Let’s consider this example class structure:

owl:Thing
   |
   |
    --> bibo:Document
            |
            |
            |
             --> bibo:Image
                     |
                     |
                     |--> muni:HeritageImage
                     |        |
                     |        |
                     |         --> muni:ParkHeritageImage
                     |
                     |
                     |--> muni:NeighborhoodImage
                     |
                     |
                     |
                      --> muni:ParkImage

  <1> a bibo:Image .
  <2> a muni:HeritageImage .
  <3> a muni:HeritageImage .
  <4> a muni:ParkHeritageImage .
  <5> a muni:ParlImage .

This class structure shows a hierarchy of images where the leaf classes are topical image classes (so classes where their individuals are considered images representing one of the topic: Heritage, Neighborhood and Park). Now let’s see how this class structure impacts Search queries, and returned results, by different tools (structSearch, structBrowse, sWebMap and others).

Here is a series of Search queries sent to a structWSF instance that has this class hierarchy loaded, using the sample specification noted above. This tables shows the results potentially returned by the Search endpoint with and without inferencing turned on:

<2> a muni:HeritageImage .
<3> a muni:HeritageImage .

<2> a muni:HeritageImage .
<3> a muni:HeritageImage .
<4> a muni:ParkHeritageImage .

<1> a bibo:Image .
<2> a muni:HeritageImage .
<3> a muni:HeritageImage .
<4> a muni:ParkHeritageImage .
<5> a muni:ParlImage .

In the Use Case #1, the user requests all of the muni:HeritageImages without inferencing. This means that the Search endpoint will return all of the records that have been typed as muni:HeritageImage. In this case, the records <2> and <3> got returned.

Use Case #2 is a variant of Use Case #1, only now with inferencing enabled. In this use case, the Search endpoint will return all the muni:HeritageImage and all the records that are typed with one of its subtypes (in this case, muni:ParkHeritageImage). For this query, records <2>, <3> and <4> got returned. This case shows where ontologies can have a dramatic impact on the system. If we modify that class hierarchy and put the muni:ParkHeritageImage as being a sub-class-of bibo:Image, then the same results would be returned for Use Case #2 than we got with Use Case #1.

With Use Case #3, the endpoint does not return any results because inferencing is disabled, and because there is no record typed as bibo:Image.

Use Case #4 is a variant of Use Case #3 where inferencing is enabled. The endpoint returns all the image records because all of them are bibo:Image by inference on type.

Filtering via the AGGR Ontology

The AGGR Ontology also has an impact on anything that displays facets of filtered searches. Amongst others, it impacts the structSearch and structBrowse conStruct modules. It also impacts different user interfaces that use the Search Web service endpoint to perform auto-completion tasks.

Tag Concepts in Text Documents

In the Open Semantic Framework, the Scones Web service endpoint is what is used to analyze unstructured text documents, then turning them into semi-structured text documents by automatically tagging concepts. The concept tagging takes place using ontology-based information extraction, or OBIE. Named entity dictionaries are the basis for entity tagging.

These concepts used for the tagging come from selected ontologies loaded on the system. The way these ontologies have been created, and the way the classes and named individuals have been defined, has a dramatic impact on the quality of the documents tagged by Scones.

Scones uses two things from ontologies:

• its classes
• its named individuals

Depending on settings, one or both of these sources may be used for scones tagging.

There are a few properties intimately related to the Scones Web service endpoint:

Help Navigate and Organize Web Portals

In OSF, ontologies also have an impact on the general organization of a Web portal and how it is navigated.

Portal Navigation

In an OSF portal, its domain ontologies use the sRelationBrowser for general navigation. The relation browser is a tool that lets users dynamically navigate a graph (that is, nodes with arcs that links these nodes). The most widespread usage of the relation browser is to let users navigate the links between ontology concepts. These concepts are the anchor points of what other content is available on the Web portal. By navigating the concepts (classes) structure, users are able to explore an OSF portal’s entire content.

Each node in the sRelationBrowser semantic component is linked to whatever other kinds of related records exist in the system. Depending on the types of those records, other semantic components can then be invoked to display this tightly related content for each node.

Ontologies thus impact navigation and discovery on an OSF portal in two ways:

1. They impact the navigation of the structure by defining which concepts are linked to other concepts and with what property
2. They define what related records may get displayed to the user based on their classes and properties.

Layouts Organization

OSF Web portals are mainly organized by Layouts. A layout is a specific page presentation format with specific design, components and ordering and sizing of those components. This page presentation is highly influenced by the kind of things indexed in the system. Generally, layouts present records of a certain type (or family of types), along with specialized functions that users are able to use to perform different actions on that set of records.

Here are a few examples of such layouts:

• Sample Chart Template
• Sample Image Template
• Sample Topic Template
• Sample WebMap Template

These layouts aggregate all of the records of a certain type (like images of all kinds), display them using different kind of tools (like an Images Gallery), and filter them depending on different filtering criteria (like mapping the position where each image got captured, on a map, within a specific neighborhood area).

The ontologies impact the general organization of the Web portal because of the kind of things that are indexed in the system interacting with the available layouts.

Manage Datasets and Ontologies

Basic settings for managing datasets and ontologies is provided by the WSF Ontology. It presently does so via two mechanisms.

Datasets Syncing Framework

The Datasets Syncing Framework behaves differently depending on the value of the wsf:crudAction property for each input record.

structOntology

The structOntology conStruct module exhibits different behavior depending on the value of the wsf:ontologyModified property for each input ontology description.

Set Access Permissions and Registrations

The WSF Ontology also has a principal purpose to describe the internal state of a structWSF instance such as the internal access control records, the datasets descriptions, the registered web service endpoints, etc. As a result, this ontology can have multiple effects on other parts of an OSF instance.

The WSF Ontology is used to describe three main areas of a structWSF installation:

1. datasets registry
2. access definition registry
3. registered web services endpoints registry

These registries are hosted in some specialized datasets in the triple store (Virtuoso for most OSF installations). The information indexed in these different registries is defined using the WSF ontology.

All structWSF Web services are affected by these registries.

It was not until recently that I found a workable process. Other people have successfully migrated Google Code SVN projects to GitHub, but I had yet to find a consolidated guide to do it. It is for this last reason that I write this blog post: to help people, if they desire, to move projects from Google Code to GitHub.

Moving from Google Code to GitHub

The protocol outlined below may appear complex, but it looks more intimidating than it really is. Moving a project takes about two to five minutes once your GitHub account and your migration computer is properly configured.

You need four things to move a Google Code SVN project to GitHub:

1. A Google Code project to move
2. A GitHub user account
3. SSH keys, and
4. A migration computer that is configured to migrate the project from Google Code to GitHub. (in this tutorial, we will use a Ubuntu server; but any other Linux/Windows/Mac computer, properly configured, should do the job)

Create GitHub Account

If you don’t already own a GitHub account, the first step is to create one here.

Create & Configure SSH Keys

Once your account has been created, you have to create and setup the SSH keys that you will use to commit the code into the Git Repository on GitHub:

1. Go to the SSH Keys Registration page of your account
2. If you already have a key, then add it to this page, otherwise read this manual to learn how to generate one

Configure Migration Server

The next step is to configure the computer that will be used to migrate the project. For this tutorial, I use a Ubuntu server to do the migration, but any Windows, Linux or Mac computer should do the job if properly configured.

The first step is to install Git and Ruby on that computer:

1	sudo apt-get install git-core git-svn ruby rubygems

To perform the migration of a Google Code SVN project to GitHub, we are using a Ruby application called svn2git that is now developed by Kevin Menard. The next step is to install svn2git on that computer:

1	sudo gem install svn2git --source http://gemcutter.org

Migrate Project

Before migrating your project, you have to link the Google Code committers to GitHub accounts. This is done by populating a simple text file that will be given as input to svn2git.

Open the authors.txt file into a temporary folder:

1	sudo vim /tmp/authors.txt

Then, for each author, you have to add the mapping between their Google Code and GitHub accounts. If a Google Code committer does not exist on GitHub, then you should map it to your own GitHub account.

1 2	(no author) = Frederick Giasson <fred@f...com>  fred@f...com = Frederick Giasson <fred@f...com>

The format of this authors.txt file is:

1	Google-Account-Username = Name-Of-Author-On-GitHub <Email-Of-Author-On-Github

Take note of the first Google Code committer (no author) mapping. This link is required for every authors.txt file. This placeholder is used to map the initial commit performed by the Google Code system. (When Google Code initializes a new project, it uses that username for creating the first commit of any project.)

When you are done, save the file.

Now that set up is complete, you are ready to migrate your project. First, let’s create the folder that will be used to checkout the SVN project on the server, and then to push it on GitHub.

1 2 3

cd /tmp/ mkdir myproject cd myproject

In this tutorial, we have a normal migration scenario. However, your migration scenario may differ from this one. It is why I would suggest you check out the different scenarios that are supported by svn2git document. Change the following command accordingly. Let’s migrate the Google Code SVN Project into the local Git repository:

1	/var/lib/gems/1.8/bin/svn2git http://myproject.googlecode.com/svn --authors /tmp/authors.txt --verbose

Make sure that no errors have been reported during the process. If it is the case, then refer to the Possible Errors and Fixes section below to troubleshoot your issue.

The next step is to create a new GitHub repository where to migrate the SVN project. Go to this GitHub page to create your new repository. Then you have to configure Git to add a remote link, from the local Git repository you created on your migration computer, to this remote GitHub repository:

1	git remote add origin git@github.com:you-github-username/myproject.git

Finally, let’s push the local Git repository master, branches and tags to GitHub. The first thing to push onto GitHub is the SVN’s trunk. It is done by running that command:

1	git push -u origin master

Then, if your project has multiple branches and tags, you can push them, one by one, using the same command. However, you will have to replace master by the name of that branch or tag. If you don’t know what is the exact name of these branches or tags, you can easily list all of them using this Git command:

1	git show-ref

Once you have progressed through all branches and tags, you are done. If you take a look at your GitHub project’s page, you should see that the trunk, branches, tags and commits are now properly imported into that project.

Possible Errors And Fixes

Fatal Error: Not a valid object name

There are a few things that can go wrong while trying to migrate your project(s).

One of the errors I experienced is a "fatal" error message "Not a valid object name". To fix this, we have to fix a line of code in svn2git. Open the migration.rb file. Check around the line 227 for the method fix_branches(). Remove the first line of that method, and replace the second one by:

1	svn_branches = @remote.find_all { |b| !@tags.include?(b) && b.strip =~ %r{^svn\/} }

Error: author not existing

While running the svn2git application, the process may finish prematurely. If you check the output, you may see that it can’t find the match for an author. What you will have to do is to add that author to your authors file and re-run svn2git. Otherwise you won’t be able to fully migrate the project.

I’m not quite sure why these minor glitches occurred during my initial migrate, but with the simple fixes above you should be good to go.

Why have Mike Bergman and I chosen to develop no less than three major open source projects (structWSF, conStruct and the Semantic Components), encompassing more than 100 000 lines of new code and leveraging between 30 to 50 other open source software and libraries? Why have we open sourced all our software? Why has open source formed the core business strategy of Structured Dynamics in the last three years? How have we been able to profitably sustain the company, even in the midst of the global economic crisis that began in 2008?

I will try to answer these questions in this blog post, perhaps even providing some guidance for newer startups that may follow behind us.

Why Open Sourcing?

Why did Structured Dynamics chose to open source all of its software? There are multiple reasons why people and businesses choose to go open source. For some, it is because they think that it is where the market place is moving. For others it is because they think that a community will emerge around their effort, and then get free resources that improve the piece of software. Some think that their software will promptly be reviewed by professional programmer. Others may think that their system will become more secure. Etc.

For Structured Dynamics the reason why we choose to go open source is somewhat different:

We perceived that by open sourcing our complete software stack we could bootstrap the company without any external investment.

Making a Living out of Open Source Projects

There are multiple ways to do a living from an open source project:

• Doing consultancy work related to the project
• Implementing the software(s) into clients’ computer environment(s)
• Selling training classes
• Selling support contracts
• Selling maintenance contracts
• Selling hosted instances of the software (the SaaS model for one)
• Selling development time to improve some part(s) of the software
• Creating conferences around their open source projects
• Selling proprietary extensions
• I am probably missing a few, so please add them in a comment section below, and I will make sure to add them to this list.

Depending on the software you are developing, and depending on the business plan of your company, you may be doing one — or multiple — of these things to generate some money from your open source projects.

At Structured Dynamics we are doing some of them: we do get consultancy contracts related to the Open Semantic Framework and we do implement OSF in our clients’ computer environments.

But, more importantly, we are also doing development contracts related to the framework. In fact, each project we are working on is quite different. Our major projects involve companies that reside in totally different domains, have different needs and need to accommodate different kinds of users. However, most of the projects share the same core needs, and all of them advance the core technology in ways meaningful to our vision. We choose our customers — and , of course, vice versa — based on a true sense of partnership wherein both parties have their objectives furthered.

Let’s see how we use these relationships to drive the development of the Open Semantic Framework.

Open Source Project as a Pool of Resources

In the last three years, Structured Dynamics has attracted multiple companies and organizations that share our vision, and which are willing to invest in the Open Semantic Framework open source project. (See Mike's recent post on business development for a bit more on that aspect of things.) Each of these clients did want to use the OSF framework for their own needs. However, each of them did want to do something special that was not currently implemented in the framework.

What we created in these three years is a pool of resources that we used to develop the framework such that it accommodates the needs of each of our clients. Each of our clients then becomes a participant to the shared pool of innovation. Our clients have been willing to invest in the open source framework because they need their own features and because they know that they will benefit from what other participants of the pool will invest themselves down the road.

In that scenario, we are the managers of a pool of resources. We have the vision of where we want the framework to go, we know the roadmap of the project and we know the needs of each participant (our clients). What we do is to try to optimize the resources we get from each of our clients by developing the framework such that it can accommodate as broad of a spectrum of participants as possible. Then, we seek to find new participants that have some needs that will help us continue to develop the next steps of the roadmap. In this manner, we Jacob's Ladder our existing work to increase the capabilities for later clients, but earlier clients still benefit because they can upgrade to the later improvements. This is a self-sustaining model to continue to move the development of the framework forward.

By finding new clients, what we do is to give a return on investment to the other pool participants. Most of the new features that we develop for these new clients will benefit the other participants to the pool and will create new possibilities for them without any additional investment. All of our first clients have implemented what other participants later invest into the pool, thus crystallizing and augmenting their return on investment by using these new features.

Open Source is Not Just About Software

Open Source is not just about pieces of code, and this is quite important to understand. What we have open sourced with the Open Semantic Framework is much more than a series of code sources. We open sourced the entire framework:

1. The source codes
2. The documentation
3. The processes
4. The methodologies

We term this comprehensive approach our total open solution.

This distinction with other open source projects is an essential differentiator with our approach. We choose to open source all of the pieces related to the framework. What drove this decision is a simple sentence that shows our philosophy behind it:

"We're Successful When We're Not Needed"

If the APIs, processes and methodologies are not properly documented, it means that we would certainly be needed by our clients, which would mean that we failed to open source our solution. But since we are working to open source our code, our processes and our methodologies, we are on the way to successfully open source the Open Semantic Framework since we won’t be needed by our clients.

This business approach is not as crazy as it sounds. We are free to work on new and important innovations, and are not basing our company culture on dependency and a constant drain by our customers. I know, it does not sound like Larry Ellison, but sounds good to us and our clients. It is certainly not a maximum revenue objective built on the backs of individual clients.

Our life is more fun and our clients trust us with new stuff. Further, each step of the way, we are able to leverage our own framework for unbelievable productivity in what we deliver for the money. But that is a topic for another day.

We think Structured Dynamics' business approach is a contemporary winning strategy. Our customers get good and advanced capabilities at low cost and risk, while we get to work on innovative extensions that are raising the semantic baseline for the marketplace. Who knows if we will always continue this path, but for now it is leading to sustained development and market growth for open semantic frameworks, including our own OSF.

In this blog post, I want to further explain their data management workflow, and how their data get exposed to different kind of users.

Two Classes of Users

Let’s take a look at their data ingest/management/publishing workflow:

However, how the data get exposed to the users is not that clear. In fact, it depends on the classes of users. A user can be multiple different things: it may be a person, it may be a computer software, it may be an organization, etc. However, there are two general classes of users:

1. Public users, and
2. Private users

Public users are users that have no direct relation with Volkswagen and that have no access to their internal network. Private users are generally internal departments or some internal software applications that have direct access to the structWSF instance.

Private Users

Private users generally have access to all structWSF web service endpoints. This means that all structWSF functionalities are accessible to them by querying the endpoints.

Two different kind of private users are specified in the use case’s schema:

1. Volkswagen Site Search
2. Other / External Applications

The Volkswagen site search is a software application that uses the structWSF Search endpoint to search, filter and expose their data to their users (the people who perform searches on the Volkswagen UK website).

These are two examples of software applications that Volkswagen created around the structWSF web services to re-purpose, re-contextualize and re-publish their RDF data.

Public Users

There is currently two kinds of public users of this new Volkswagen Semantic Platform:

• People, and
• Software applications

Two interfaces have been made publicly available for each of these kinds of users:

• A website search engine page for people, and
• A SPARQL endpoint for software applications

When a person user reaches the website’s search page, the search query get sent to the structWSF Search web service endpoint. The result is then returned to the engine, get templated and displayed to the person user.

A SPARQL endpoint is accessible to the software applications. This endpoint is hosted by the Kasabi information marketplace. Volkswagen chooses to export everything from their structWSF into Kasabi to outsource the maintenance of their public SPARQL endpoint.

Unlock the Power

As we saw in this blog post and in the W3C use case, all Volkswagen UK data is internally managed by structWSF; however they are not locked into that system. They can easily communicate with external services to add new functionalities to their stack or to take business decision such as outsourcing the management of some publicly accessible data access endpoints.

This is an important characteristic of their design:

By choosing semantic web technologies (such as structWSF), techniques and concepts (such as their Vehicles OWL Ontology and RDF), they are not locking themselves into a specific framework. They can easily communicate with external systems and applications. This means that they can quickly adapt their system to their constantly changing needs.

Conclusion

I wrote this blog post to further explain Volkswagen’s data management workflow. I wanted to make sure that people were understanding the role that structWSF has in this use case, and the ecosystem it operates in.