Ontologies are to the Open Semantic Framework what humans were to the Mechanical Turk. The hidden human in the Mechanical Turk was orchestrating all and every chess move. However, to the observers, the automated chess machine was looking just like it: a new kind of intelligent machine. We were in 1770.

Ontologies plays exactly the same role for the Open Semantic Framework (OSF): they orchestrate all and every moves for all the pieces within OSF. They are what instructs structWSF, the Semantic Components, conStruct, and all other derivate pieces of user interfaces how to behave.

In this (lengthy) blog post, I will present the main ontologies that have an impact on different parts of OSF. We will see how different ontology classes and properties, and how the description of the records indexed in the system, can impact the behaviors of OSF.

In addition to this post, Mike has also published a blog post today that overviews the overall OSF ontology modularization and architecture.

Constituent Ontologies

Let’s take a look at the core ontologies used by the Open Semantic Framework. All these ontologies have been developed in relation to OSF. These, and other external ontologies, have the same role in OSF as the human does in the Mechanical Turk: they instruct the system how to behave.

Here is the list of the core ontologies:

1. The SCO Ontology (Semantic Component Ontology)
2. The WSF Ontology (Web Service Framework Ontology)
3. The AGGR Ontology (Aggregation Ontology)
4. The irON Ontology (Instance Record an Object Notation Ontology)
5. One or more domain ontologies, to capture the concepts and relationships for the purposes of a given OSF installation, and
6. Possibly UMBEL or other upper-level concept ontologies, used for linkages to external systems.

(Note: the internal wiki links to each of these ontologies also provides links to the actual ontology specifications on Github.)

A useful discussion of these ontologies and their interactions in an OSF instance is provided by the ontology modularization document. This current document focuses primarily on the specific properties and roles associated with them in an OSF installation.

Depending on the specific OSF installation, of course, multiple external ontologies may also be employed. Some of the common external ones used in an OSF installation are described by the external ontologies document. These external ontologies are important — indeed essential in order to ensure linkage to the external world — but have little to do with internal OSF control structures. That is why the rest of this discussion focused on internal ontologies only.

Summary Ontology Roles

Ontologies play pivotal roles across all parts of the framework. In a broad sense, the internal OSF ontologies are used for annotations, guiding interactions or relating concepts and information to other information. In specific terms, OSF ontologies may play one or more of these dozen or so roles:

1. Define record descriptions
2. Inform interface displays
3. Integrate different data sources
4. Define component selections
5. Define component behaviors
6. Guide template selection
7. Provide reasoning and inference
8. Guide content filtering (with and without inference)
9. Tag concepts in text documents
10. Help organize and navigate Web portals
11. Manage datasets and ontologies, and
12. Set access permissions and registrations.

In the remainder of this post, for each of these roles, we will see how ontologies affect numerous different parts of the OSF framework. These sections are presented in the order above.

Define Records Descriptions

A central role of ontologies in the Open Semantic Framework is their use to describe any kind of record that gets indexed and managed by the system. Since the framework indexes everything into the RDF data model, ontologies are needed as a schema to describe these RDF resources.

The irON ontology is specifically designed for record descriptions and notations. It interacts with all of the domain and (if used, UMBEL) upper level ontologies.

Inform Interface Displays

Ontologies have an impact in most of the user interfaces that display record information. The property that has the most impact is iron:prefLabel, which is used to display the label within the user interface that refers to a record or record attributes (properties). This label can be used within text, in a list control, in a tree control, or in any other kind of control that displays references to records.

Note: there are also other properties that are considered as fallbacks to iron:prefLabel if a record has no triples using the iron:prefLabel property. These include rdfs:label, dcterms:title, foaf:name, etc.

General User Interface Labels and Descriptions

There are a few properties that have an impact on most of the components of the OSF stack, most of which come from the irON ontology. Here is the list of these irON properties that impact other parts of the system, mainly related to different user interfaces:

User Interface ‘Short’ Labels

There are a few properties that impact most of the components of the OSF stack. Here is the list of SCO properties that impact other parts of the system, mainly related to different user interfaces:

Hierarchical Displays

The way ontologies define a class or a property structure also has an impact on different kinds of hierarchical displays. An example of this is the “Filter by Kinds” section of the structSearch and structBrowse modules. The possible filters that may be applied to a search query will be displayed to the user according to the hierarchy as defined in the ontologies.

Integrate Heterogeneous Data Sources

The principle reason why the Open Semantic Framework uses RDF and ontologies to describe all the data it indexes and manages is to facilitate data integration from multiple and heterogeneous data sources. The premise of using RDF and ontologies is:

This foundation leads to the extreme flexibility of the Open Semantic Framework. The rationale behind this flexibility, and its benefits, has been described in many locations within this wiki. You may also want to see this article on One of the Semantic Web's Core Added Value.

Ontologies have a dramatic — and positive — impact on the data integration and presentation tasks within an OSF instance.

Define Component Selections

A key aspect of the SCO Ontology is its use as the means to define what semantic components (or widgets) display what types of information within data records.

These assignments are done via the sControl component. The properties for this component define what components may display what type (class) of data records. Here is the list of SCO properties that impact the sControl’s behaviors:

Define Component Behaviors

In the Open Semantic Framework, one of the most important roles of ontologies is to define the interaction between different pieces of the system. Because of the extent of these interactions, this section is the longest and most detailed amongst all of the dozen or so ontology roles.

The SCO ontology can have multiple effects on multiple parts of an OSF instance. This section describes those interactions.

sMap Component

The sMap component had different behaviors depending on how its input record is described. Here is the list of SCO properties that will have an impact on the sMap’s behaviors:

sWebMap Component

The sWebMap component has different behaviors depending on how its input record is described. Here is the list of SCO and WGS84 properties that impact the behavior of an sWebMap:

sStory Component

The sStory component has different behaviors depending on how its input record is described. Here is the list of SCO properties that impact an sStory component:

sBarChart and sLinearChart Components

The sBarChart and the sLinearChart components exhibit different behaviors depending on how the input records that are enabled for these component types are described. Here is the list of the SCO properties that impact this behavior:

sRelationBrowser

The sRelationBrowser component exhibits different behaviors depending on how its input record is described. Here is the list of SCO properties that impact the sRelationBrowser component:

sDashboard

The sDashboard component exhibits different behaviors depending on how its input record is described. Here is the list of SCO properties that impact the sDashboard component:

Guide Visualization Template Selection

One of the core features of the conStruct set of Drupal modules is the ability to use different display templates depending on the types of records available. The selection of these templates is based on the types of those records and the type hierarchies described by the OSF ontologies. This section describes how these ontologies guide template selections.

As a refresher on templates and their use, see the Building conStruct Templates document. It describes how the templating engine works and how to create various templates.

Template Selection

Template selection is the action of binding an instance record to a display template based on its type. Three things are required to make this happen:

1. Instance records have to be typed
2. An ontological structure of type relationships (via subClasses) has to exist in one or more OSF ontology(ies), and
3. A template has to exist for the type of the instance record.

(Note: a specific template by type is not strictly required, since lacking a specific template for the target type, the system will invoke the nearest template up the parental chain in the governing ontology structure, eventually getting to the most generic template available, that for “thing”.)

Impact of Ontologies on Template Selection

conStruct’s templating engine selects record display templates based on the class hierarchy loaded on a OSF instance. It also uses inference on types to select the proper template for a given record.

Let’s say that we try to display information about a foaf:Person instance record. What the system attempts to do is to find a template that displays information about this kind of instance record. First, the foaf:Person type (class) has to be defined in the ontological structure of the OSF instance; if it is not, then no specific template will be selected and the system will default to using the owl_thing.html template (see below). If the type (class) is found, the system will next check to see if a template exists for that specific type. If one exists one, it will use the matching template. If one does not, it will next select the parent class of the type and try the match again. If it again fails, it will continue its test up the parental chain. If all tests fail, it will use the default owl_thing.html template. Whichever template is selected then becomes the basis for formatting and presenting the visual record display.

We can use a simple class hierarchy, matched to a simple set of available template files, to illustrate how ontologies impact the conStruct templating system.

owl:Thing
   |
   |
    --> foaf:Agent
            |
            |
            |--> foaf:Person
            |
            |
             --> foaf:Organization

  owl_thing.tpl
  foaf_agent.tpl
  foaf_organization.tpl

Now, let’s say that our OSF portal is about to display information about a foaf:Person record. As we can notice, there is no foaf_person.tpl template available for a foaf:Person. However, because of the ontology structure, the system next attempts to select a template from a parent class of that foaf:Person.

What the system would do is to check if there is a template available for a record of type foaf:Person. Since there is none, it would try to find one for a parent type, so in this case the foaf:Agent class. In our example, there is now a match. The templating engine thus uses the foaf_agent.tpl template to display information about the foaf:Person record.

Were the foaf_agent.tpl not to exist, then the templating engine would fall back to the owl_thing.tpl template, which is considered to be the “generic record display template”, or the template of last resort.

This design means that if:

1. the ontological structure changes over time, or
2. new templates get added to the system

then there may be an impact on how the record gets displayed.

The major advantage of this design is that more and more specific formatting templates may be added to an OSF installation over time, both improving the tailored look of results displays and accommodating more structure and relationships as they evolve.

Provide Reasoning and Inference

A standard use of ontologies is for reasoning and inference, and those used by OSF are no exception.

By extension, however, we can also use these same capabilities to check on data consistency and coherence. This is an important feature of the system since the system can detect if there are logical inconsistencies or logical incoherencies that have been developed by the system administrator during ontology growth and development. Having coherent and consistent ontologies means that we have the proper foundations to create consistent and coherent datasets of instance records.

See further the discussion on reasoning using Protégé.

Guide Content Filtering

Filtering data is the action of getting a subset of records from a complete dataset based on some selection criteria. In OSF, the predominant share of filtering is done using the structWSF Search Web service endpoint. The a minority of filtering is done using the SPARQL endpoint. It is also possible to filter via the AGGR aggregation ontology.

Possible filtering criteria for the Search endpoint are:

1. Filtering by type(s)
2. Filtering by attribute(s)
3. Filtering by attribute(s)/value(s)
4. Filtering by geo-localization (within a given geographical area)

These filtering activities are performed by different tools of the stack, such as:

• structSearch
• structBrowse
• sWebMap

These tools are impacted by the definition of the loaded ontologies. The filtering of the values by types, attributes and attributes/values requires an ontology class or an ontology property as filtering criteria.

Filtering with Inference

Also, the any Search query can be performed with inference enabled. Just like with the template selection section noted above, inference can have a big impact on the number and nature of returned results. Let’s consider this example class structure:

owl:Thing
   |
   |
    --> bibo:Document
            |
            |
            |
             --> bibo:Image
                     |
                     |
                     |--> muni:HeritageImage
                     |        |
                     |        |
                     |         --> muni:ParkHeritageImage
                     |
                     |
                     |--> muni:NeighborhoodImage
                     |
                     |
                     |
                      --> muni:ParkImage

  <1> a bibo:Image .
  <2> a muni:HeritageImage .
  <3> a muni:HeritageImage .
  <4> a muni:ParkHeritageImage .
  <5> a muni:ParlImage .

This class structure shows a hierarchy of images where the leaf classes are topical image classes (so classes where their individuals are considered images representing one of the topic: Heritage, Neighborhood and Park). Now let’s see how this class structure impacts Search queries, and returned results, by different tools (structSearch, structBrowse, sWebMap and others).

Here is a series of Search queries sent to a structWSF instance that has this class hierarchy loaded, using the sample specification noted above. This tables shows the results potentially returned by the Search endpoint with and without inferencing turned on:

<2> a muni:HeritageImage .
<3> a muni:HeritageImage .

<2> a muni:HeritageImage .
<3> a muni:HeritageImage .
<4> a muni:ParkHeritageImage .

<1> a bibo:Image .
<2> a muni:HeritageImage .
<3> a muni:HeritageImage .
<4> a muni:ParkHeritageImage .
<5> a muni:ParlImage .

In the Use Case #1, the user requests all of the muni:HeritageImages without inferencing. This means that the Search endpoint will return all of the records that have been typed as muni:HeritageImage. In this case, the records <2> and <3> got returned.

Use Case #2 is a variant of Use Case #1, only now with inferencing enabled. In this use case, the Search endpoint will return all the muni:HeritageImage and all the records that are typed with one of its subtypes (in this case, muni:ParkHeritageImage). For this query, records <2>, <3> and <4> got returned. This case shows where ontologies can have a dramatic impact on the system. If we modify that class hierarchy and put the muni:ParkHeritageImage as being a sub-class-of bibo:Image, then the same results would be returned for Use Case #2 than we got with Use Case #1.

With Use Case #3, the endpoint does not return any results because inferencing is disabled, and because there is no record typed as bibo:Image.

Use Case #4 is a variant of Use Case #3 where inferencing is enabled. The endpoint returns all the image records because all of them are bibo:Image by inference on type.

Filtering via the AGGR Ontology

The AGGR Ontology also has an impact on anything that displays facets of filtered searches. Amongst others, it impacts the structSearch and structBrowse conStruct modules. It also impacts different user interfaces that use the Search Web service endpoint to perform auto-completion tasks.

Tag Concepts in Text Documents

In the Open Semantic Framework, the Scones Web service endpoint is what is used to analyze unstructured text documents, then turning them into semi-structured text documents by automatically tagging concepts. The concept tagging takes place using ontology-based information extraction, or OBIE. Named entity dictionaries are the basis for entity tagging.

These concepts used for the tagging come from selected ontologies loaded on the system. The way these ontologies have been created, and the way the classes and named individuals have been defined, has a dramatic impact on the quality of the documents tagged by Scones.

Scones uses two things from ontologies:

• its classes
• its named individuals

Depending on settings, one or both of these sources may be used for scones tagging.

There are a few properties intimately related to the Scones Web service endpoint:

Help Navigate and Organize Web Portals

In OSF, ontologies also have an impact on the general organization of a Web portal and how it is navigated.

Portal Navigation

In an OSF portal, its domain ontologies use the sRelationBrowser for general navigation. The relation browser is a tool that lets users dynamically navigate a graph (that is, nodes with arcs that links these nodes). The most widespread usage of the relation browser is to let users navigate the links between ontology concepts. These concepts are the anchor points of what other content is available on the Web portal. By navigating the concepts (classes) structure, users are able to explore an OSF portal’s entire content.

Each node in the sRelationBrowser semantic component is linked to whatever other kinds of related records exist in the system. Depending on the types of those records, other semantic components can then be invoked to display this tightly related content for each node.

Ontologies thus impact navigation and discovery on an OSF portal in two ways:

1. They impact the navigation of the structure by defining which concepts are linked to other concepts and with what property
2. They define what related records may get displayed to the user based on their classes and properties.

Layouts Organization

OSF Web portals are mainly organized by Layouts. A layout is a specific page presentation format with specific design, components and ordering and sizing of those components. This page presentation is highly influenced by the kind of things indexed in the system. Generally, layouts present records of a certain type (or family of types), along with specialized functions that users are able to use to perform different actions on that set of records.

Here are a few examples of such layouts:

• Sample Chart Template
• Sample Image Template
• Sample Topic Template
• Sample WebMap Template

These layouts aggregate all of the records of a certain type (like images of all kinds), display them using different kind of tools (like an Images Gallery), and filter them depending on different filtering criteria (like mapping the position where each image got captured, on a map, within a specific neighborhood area).

The ontologies impact the general organization of the Web portal because of the kind of things that are indexed in the system interacting with the available layouts.

Manage Datasets and Ontologies

Basic settings for managing datasets and ontologies is provided by the WSF Ontology. It presently does so via two mechanisms.

Datasets Syncing Framework

The Datasets Syncing Framework behaves differently depending on the value of the wsf:crudAction property for each input record.

structOntology

The structOntology conStruct module exhibits different behavior depending on the value of the wsf:ontologyModified property for each input ontology description.

Set Access Permissions and Registrations

The WSF Ontology also has a principal purpose to describe the internal state of a structWSF instance such as the internal access control records, the datasets descriptions, the registered web service endpoints, etc. As a result, this ontology can have multiple effects on other parts of an OSF instance.

The WSF Ontology is used to describe three main areas of a structWSF installation:

1. datasets registry
2. access definition registry
3. registered web services endpoints registry

These registries are hosted in some specialized datasets in the triple store (Virtuoso for most OSF installations). The information indexed in these different registries is defined using the WSF ontology.

All structWSF Web services are affected by these registries.

Last week we slowly migrated Structured Dynamics‘ Google Code Projects to GitHub.We have been thinking about moving to GitHub for some time now, but we only wanted to move projects to it if no prior history and commits were dropped in the process. One motivation for the possible change has been the seeming lack of support by Google for certain long-standing services: we are seeing disturbing trends across a number of existing services. We also needed a migration process that would work with all of our various projects, without losing a trunk, branch, tag or commits (and their related comments).

It was not until recently that I found a workable process. Other people have successfully migrated Google Code SVN projects to GitHub, but I had yet to find a consolidated guide to do it. It is for this last reason that I write this blog post: to help people, if they desire, to move projects from Google Code to GitHub.

Moving from Google Code to GitHub

The protocol outlined below may appear complex, but it looks more intimidating than it really is. Moving a project takes about two to five minutes once your GitHub account and your migration computer is properly configured.

You need four things to move a Google Code SVN project to GitHub:

1. A Google Code project to move
2. A GitHub user account
3. SSH keys, and
4. A migration computer that is configured to migrate the project from Google Code to GitHub. (in this tutorial, we will use a Ubuntu server; but any other Linux/Windows/Mac computer, properly configured, should do the job)

Create GitHub Account

If you don’t already own a GitHub account, the first step is to create one here.

Create & Configure SSH Keys

Once your account has been created, you have to create and setup the SSH keys that you will use to commit the code into the Git Repository on GitHub:

1. Go to the SSH Keys Registration page of your account
2. If you already have a key, then add it to this page, otherwise read this manual to learn how to generate one

Configure Migration Server

The next step is to configure the computer that will be used to migrate the project. For this tutorial, I use a Ubuntu server to do the migration, but any Windows, Linux or Mac computer should do the job if properly configured.

The first step is to install Git and Ruby on that computer:

1	sudo apt-get install git-core git-svn ruby rubygems

To perform the migration of a Google Code SVN project to GitHub, we are using a Ruby application called svn2git that is now developed by Kevin Menard. The next step is to install svn2git on that computer:

1	sudo gem install svn2git --source http://gemcutter.org

Migrate Project

Before migrating your project, you have to link the Google Code committers to GitHub accounts. This is done by populating a simple text file that will be given as input to svn2git.

Open the authors.txt file into a temporary folder:

1	sudo vim /tmp/authors.txt

Then, for each author, you have to add the mapping between their Google Code and GitHub accounts. If a Google Code committer does not exist on GitHub, then you should map it to your own GitHub account.

1 2	(no author) = Frederick Giasson <fred@f...com>  fred@f...com = Frederick Giasson <fred@f...com>

The format of this authors.txt file is:

1	Google-Account-Username = Name-Of-Author-On-GitHub <Email-Of-Author-On-Github

Take note of the first Google Code committer (no author) mapping. This link is required for every authors.txt file. This placeholder is used to map the initial commit performed by the Google Code system. (When Google Code initializes a new project, it uses that username for creating the first commit of any project.)

When you are done, save the file.

Now that set up is complete, you are ready to migrate your project. First, let’s create the folder that will be used to checkout the SVN project on the server, and then to push it on GitHub.

1 2 3

cd /tmp/ mkdir myproject cd myproject

In this tutorial, we have a normal migration scenario. However, your migration scenario may differ from this one. It is why I would suggest you check out the different scenarios that are supported by svn2git document. Change the following command accordingly. Let’s migrate the Google Code SVN Project into the local Git repository:

1	/var/lib/gems/1.8/bin/svn2git http://myproject.googlecode.com/svn --authors /tmp/authors.txt

Make sure that no errors have been reported during the process. If it is the case, then refer to the Possible Errors and Fixes section below to troubleshoot your issue.

The next step is to create a new GitHub repository where to migrate the SVN project. Go to this GitHub page to create your new repository. Then you have to configure Git to add a remote link, from the local Git repository you created on your migration computer, to this remote GitHub repository:

1	git remote add origin git@github.com:you-github-username/myproject.git

Finally, let’s push the local Git repository master, branches and tags to GitHub. The first thing to push onto GitHub is the SVN’s trunk. It is done by running that command:

1	git push -u origin master

Then, if your project has multiple branches and tags, you can push them, one by one, using the same command. However, you will have to replace master by the name of that branch or tag. If you don’t know what is the exact name of these branches or tags, you can easily list all of them using this Git command:

1	git show-ref

Once you have progressed through all branched and tags, you are done. If you take a look at your GitHub project’s page, you should see that the trunk, branches, tags and commits are now properly imported into that project.

Possible Errors And Fixes

Fatal Error: Not a valid object name

There are a few things that can go wrong while trying to migrate your project(s).

One of the errors I experienced is a "fatal" error message "Not a valid object name". To fix this, we have to fix a line of code in svn2git. Open the migration.rb file. Check around the line 227 for the method fix_branches(). Remove the first line of that method, and replace the second one by:

1	svn_branches = @remote.find_all { |b| !@tags.include?(b) && b.strip =~ %r{^svn\/} }

Error: author not existing

While running the svn2git application, the process may finish prematurely. If you check the output, you may see that it can’t find the match for an author. What you will have to do is to add that author to your authors file and re-run svn2git. Otherwise you won’t be able to fully migrate the project.

I’m not quite sure why these minor glitches occurred during my initial migrate, but with the simple fixes above you should be good to go.

Two Main structWSF Characteristics: Accessibility & Management

structWSF is a set of 22 web service endpoints that lets you integrate data from different sources, manage that integrated data, and publish it via different communication channels such as web pages, software applications, etc.

Obviously, the main characteristic of this framework is that everything is a web service. This means that all functionality of the system can be accessed from anywhere on the Internet. However, this doesn't mean that everything is open like a snack-bar. In fact, there are two levels of accessibility: (1) access to the web service endpoint's URL, and (2) access to the content of datasets hosted on structWSF. Depending on the usecase, people could restrict the direct access to the web service endpoint(s) by properly configuring their web server, others could choose to let anyone access the endpoints, but would restrict the access to the dataset(s) hosted by structWSF. In case of MyPeg.ca, the sponsor chose to open the access to their web service endpoints and datasets.

Just by surfing on the MyPeg.ca portal, you are already leveraging these endpoints in multiple different ways. First, each time you generate a browse or a search Web page, you are telling the web server to send multiple queries to different endpoints; then the web page's content will be populated with that information and presented to you. But, each time you click on an explorer node, your web browser is also sending queries to exactly the same web service endpoints. So, in one case a PHP script acts to query the endpoints; and, in other cases, a Flash Semantic Component does. Depending, all structWSF data can be accessed from quite different environments.

The other main characteristic of structWSF is that any kind of data can be imported in, and exported out, of the system. structWSF leverages RDF (Resource Description Framework) as the canonical data format that can be used to express any other formats. It is because of the usage of RDF that structWSF can act as an effective ETL (Extract, transform, load) system. Depending on the web service endpoint, the output formats currently supported by most of the endpoints are:

• text/xml (structXML)
• application/json (structXML in JSON)
• application/iron+json (irJSON)
• application/rdf+xml (RDF/XML)
• application/rdf+n3 (RDF/N3)

But the architecture of the web service endpoints can easily accommodate other formats if needed for a specific usecase.

Getting Data Out Of MyPeg.ca

Now, how can you get data out of MyPeg.ca? There are really two methods. This blog post discusses the CRUD: Read, Browse and Search web service endpoints. In my next blog post, I will focus on using the SPARQL web service endpoint to do the same.

All of the query examples in this blog post will use a tool called Curl to send the queries and to get back the resultsets. I encourage you to download and use that tool to test these endpoints and to gain a feeling for how it works. Also note that only the first record of each resultset is recorded below (of course, the actual results include all records).

Browse

The Browse web service endpoint is used to return lists of records. These records can also be filtered according to their provenance (dataset), type and the attributes that describe them. Now, let's see how you can use this web service to get data out of MyPeg.ca.

First, there are three datasets available to the public:

1. Well-being Indicators (http://www.mypeg.ca/wsf/datasets/258/)
2. Stories (http://www.mypeg.ca/wsf/datasets/272/)
3. PEG Framework (http://www.mypeg.ca/wsf/datasets/249/)

The resultsets can be serialized using one of these four different formats:

• text/xml (structXML)
• application/json (structXML in JSON)
• application/rdf+xml (RDF/XML)
• application/rdf+n3 (RDF/N3)

Note: if one of your desired formats is not directly available at the endpoint level, you can always use one of the converter web service endpoints such as: commON, irJSON or TSV/CSV.

Get the first 10 results of the Stories dataset in structXML

Query:

curl -H "Accept: text/xml" "http://www.mypeg.ca/ws/browse/" -d "attributes=all&types=all&datasets=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2F&items=10&page=0&inference=on&include_aggregates=true"

StructXML resultset:

<?xml version="1.0" encoding="utf-8"?>
<resultset>
<prefix entity="owl" uri="http://www.w3.org/2002/07/owl#"/>
<prefix entity="rdf" uri="http://www.w3.org/1999/02/22-rdf-syntax-ns#"/>
<prefix entity="rdfs" uri="http://www.w3.org/2000/01/rdf-schema#"/>
<prefix entity="wsf" uri="http://purl.org/ontology/wsf#"/>
<subject type="http://purl.org/ontology/muni#Story" uri="http://www.mypeg.ca/wsf/datasets/272/resource/AgeOpportunity">
<predicate type="http://purl.org/dc/terms/isPartOf">
<object type="http://rdfs.org/ns/void#Dataset" uri="http://www.mypeg.ca/wsf/datasets/272/"/>
</predicate>
<predicate type="http://purl.org/ontology/iron#prefLabel">
<object type="rdfs:Literal">Age &amp; Opportunity</object>
</predicate>
<predicate type="http://purl.org/dc/terms/created">
<object type="rdfs:Literal">2010-10-28T19:38:58+00:00</object>
</predicate>
<predicate type="http://purl.org/ontology/bibo/abstract">
<object type="rdfs:Literal">Amanda Macrae, Deborah Lorteau and Stacey Miller work for Age and Opportunity.
The majority of clients are older adults living at lower socio economic status. When addressing the housing issue they say, "In a nutshell, it's dire." There is simply not enou...</object>
</predicate>
<predicate type="http://purl.org/ontology/peg#interviewee">
<object type="rdfs:Literal">Amanda Macrae, Deborah Lorteau, Stacey Miller</object>
</predicate>
<predicate type="http://purl.org/ontology/peg#interviewer">
<object type="rdfs:Literal">Molly Johnson</object>
</predicate>
<predicate type="http://purl.org/ontology/peg#storyRelatedAgencyProgram">
<object type="rdfs:Literal">Age &amp; Opportunity</object>
</predicate>
<predicate type="http://purl.org/ontology/sco#storyAnnotatedTextUri">
<object>http://www.mypeg.ca/scones/AgeOpportunity.xml</object>
</predicate>
<predicate type="http://purl.org/ontology/sco#storyTextUri">
<object type="rdfs:Literal">http://www.mypeg.ca/scones/AgeOpportunity.txt</object>
</predicate>
</subject>
</resultset>

Get the 10 first results from all datasets that are records of type Neighborhoods in RDF/XML

Query:

curl -H "Accept: application/rdf+xml " "http://www.mypeg.ca/ws/browse/" -d "attributes=all&amp; type=http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23Neighborhood &amp;datasets=all&amp;items=10&amp;page=0&amp;inference=on&amp;include_aggregates=true"

RDF/XML resultset:

<?xml version="1.0"?>
<rdf:RDF  xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:wsf="http://purl.org/ontology/wsf#" xmlns:ns0="http://purl.org/ontology/peg#" xmlns:ns1="http://purl.org/dc/terms/" xmlns:ns2="http://purl.org/ontology/iron#" xmlns:ns3="" xmlns:ns4="http://purl.org/dc/elements/1.1/" xmlns:ns5="http://purl.org/ontology/aggregate#">

<ns0:Component rdf:about="http://purl.org/ontology/peg/framework#Safety">
<ns1:isPartOf rdf:resource="http://www.mypeg.ca/wsf/datasets/249/" />
<ns2:prefLabel>Safety</ns2:prefLabel>
<ns2:altLabel>safety</ns2:altLabel>
<ns3:>safety</ns3:>
<ns4:description>Safety is the state of being "safe", the condition of being protected against physical, social, spiritual, financial, political, emotional, occupational, psychological, educational or other types or consequences of failure, damage, error, accidents, harm or any other event which could be considered non-desirable.</ns4:description>
<rdfs:comment>Includes the idea of safety prevention</rdfs:comment>
<rdfs:seeAlso>http://en.wikipedia.org/wiki/Safety</rdfs:seeAlso>
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#HouseholdIncome" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#LowIncomeCutOffAfterTax" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#MarketBasketMeasure" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#ParticipationInSportsAndRecreation" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#MaternalSocialIsolation" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#PersonalSafety" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#EarlyDevelopmentInstrument" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#HighSchoolGraduationRate" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#LongTermUnemployment" />
<ns0:hasIndicator rdf:resource="http://purl.org/ontology/peg/framework#TeenageBirths" />
<ns0:isComponentOf rdf:resource="http://purl.org/ontology/peg/framework#BasicNeeds" />
<ns0:isComponentOf rdf:resource="http://purl.org/ontology/peg/framework#Poverty" />
</ns0:Component>
</rdf:RDF>

Search

The Search web service endpoint is also used to return lists of records. These records should match a search string and can also be filtered according to their provenance (dataset), type and the attributes that describe them.

The same mime types and datasets as the ones for the Browse web service are available for the Search endpoint.

Searching for records with the keyword "poverty" and get resultsets in RDF/N3

Query:

curl -H "Accept: application/rdf+n3" "http://www.mypeg.ca/ws/search/" -d "query=poverty&amp;datasets=all&amp;items=10&amp;page=0&amp;inference=on&amp;include_aggregates=true"

RDF/N3 resultset:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix wsf: <http://purl.org/ontology/wsf#> .

<http://purl.org/ontology/peg/framework#Poverty> a <http://purl.org/ontology/peg#CrossCuttingIssue> ;
<http://purl.org/dc/terms/isPartOf> <http://www.mypeg.ca/wsf/datasets/249/> ;
<http://purl.org/ontology/iron#prefLabel> """Poverty""" ;
<http://purl.org/dc/elements/1.1/description> """Poverty is not having the sufficient resources, capabilities, choices, security and power necessary to enjoy an adequate standard of living.  Poverty includes much more than a lack of money.  It includes being excluded from ordinary living patterns, customs and activities.  Consequently, people living in poverty are often unable to participate fully in their communities or to reach their full potential.""" ;
<http://www.w3.org/2000/01/rdf-schema#seeAlso> """http://en.wikipedia.org/wiki/Poverty""" .

CRUD: Read

The Browse and Search web service endpoints are really used to find lists of records according to some provided criteria. However, the complete description of these records is not returned by these endpoints, but only the information necessary to create the proper list to display to users in a user interface. So, to get the complete description of a record (or multiples thereof), you have to use the CRUD: Read web service endpoint. Also, sometimes you may get a reference to a record hosted on a structWSF node, then CRUD: Read is the way to get its full description.

Get the full description of the Ida story in irJSON

Query:

curl -H "Accept: application/iron+json" "http://www.mypeg.ca/ws/crud/read/?uri=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2Fresource%2FIda&amp;dataset=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2F&amp;include_reification=true&amp;include_linksback=false

irJSON resulset:

{
"dataset": {
"linkage": [
{
"linkedType": "application/rdf+xml",
"attributeList": {
"created": {
"mapTo": "http://purl.org/dc/terms/created"
},
"isAbout": {
"mapTo": "http://umbel.org/umbel#isAbout"
},
"prefLabel": {
"mapTo": "http://purl.org/ontology/iron#prefLabel"
},
"interviewee": {
"mapTo": "http://purl.org/ontology/peg#interviewee"
},
"interviewer": {
"mapTo": "http://purl.org/ontology/peg#interviewer"
},
"abstract": {
"mapTo": "http://purl.org/ontology/bibo/abstract"
},
"storyVideoAudio": {
"mapTo": "http://purl.org/ontology/peg#storyVideoAudio"
},
"storyAnnotatedTextUri": {
"mapTo": "http://purl.org/ontology/sco#storyAnnotatedTextUri"
},
"storyTextUri": {
"mapTo": "http://purl.org/ontology/sco#storyTextUri"
}
},
"typeList": {
"Story": {
"mapTo": "http://purl.org/ontology/muni#Story"
}
}
}
]},
"recordList": [
{
"id": "http://www.mypeg.ca/wsf/datasets/272/resource/Ida",
"type": "Story",
"created": "2010-10-28T18:11:27+00:00",
"isAbout": [
{
"ref": "@@http://purl.org/ontology/peg/framework#EducationAndLearning"
},
{
"ref": "@@http://purl.org/ontology/peg/framework#Health"
},
{
"ref": "@@http://purl.org/ontology/peg/framework#Program"
},
{
"ref": "@@http://purl.org/ontology/peg/framework#Income"
},
{
"ref": "@@http://purl.org/ontology/peg/framework#Poverty"
}     ],
"prefLabel": "Ida",
"interviewee": "Ida",
"interviewer": "Christa Rust",
"abstract": "'Poverty is earning just enough to get by; never having money for extras.'\n\nIda is the mother of two grown children, eight years apart.  She lives in a small bachelor suite, which costs her $428 per month, or 62% of her income.  She volunteers twice a we...",
"storyVideoAudio": "http://www.youtube.com/v/0zIqtYhiHfM",
"storyAnnotatedTextUri": "http://www.mypeg.ca/scones/Ida.xml",
"storyTextUri": "http://www.mypeg.ca/scones/Ida.txt"
}
]
}

Get Well-Being record description with linkbacks in RDF+N3

The characteristic of this query is that I enabled the "include_linksback" parameter. This returns a reference to all the records, in the datasets hosted on the structWSF node, that refers to that target record.

Query:

curl -H "Accept: application/rdf+n3" "http://www.mypeg.ca/ws/crud/read/?uri=http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing&amp;datast=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F249%2F&amp;registered_ip=self%3A%3A0&amp;include_reification=true&amp;include_linksback=true"

RDF+N3 resultset:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

<http://purl.org/ontology/peg/framework#WellBeing> a <http://purl.org/ontology/peg/framework#WellBeing> ;
<http://purl.org/ontology/iron#prefLabel> """Well-being""" ;
<http://purl.org/dc/elements/1.1/description> """Well-being refers to the general quality of life experienced by individuals and communities. The elements of wellbeing include: the ability to meet basic needs, the economy, health, the built environment, governance, education and learning, the natural environment, and social vitality.""" ;
<http://purl.org/ontology/sco#displayComponent> <http://purl.org/ontology/sco#sRelationBrowser> .

<http://purl.org/ontology/peg/framework#WellBeing> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Economy> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Governance> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#BuiltEnvironment> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#NaturalEnvironment> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#SocialVitality> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#BasicNeeds> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#EducationAndLearning> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Health> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

General Endpoint Parameters

The general parameters available for each of these web services is provided in their respective TechWiki documentation. For that detailed information, see the Browse, Search, or CRUD: Read articles.

Conclusion

As you can see, agents can get different kinds of data from the MyPeg.ca portal by querying a set of web service endpoints. This is one way to get data out of the system. These data can then be accessed for indexing in other systems, for direct use, or for dynamic applications like browsing the nodes in the explorer.

This is one of the ways to get data out of the system. A user can also export that same information from the Export features on the Browse, Search and Record View pages. Also, other methods will be explained in the next blog posts from this MyPeg.ca series.

All in all, this shows how effective structWSF can be to integrate, manage and publish a wide range of data in different data formats. It also shows how completely different parts of your software architecture can leverage your information, the way you want, from anywhere on the Internet.

Now that the MyPeg.ca project has been unveiled at the Winnipeg Poverty Reduction Partnership Forum, I can now start to write about each and every feature of this innovative website.

MyPeg.ca is a public indicators Web portal for the Canadian city of Winnipeg. It is supported by an open-source semantic web framework called OSF. This initial beta version of the Web portal emphasizes the integration, management, exploration and display of a few hundred Well-being indicators’ data for the city.

This community indicators portal is currently the best example of a Citizen Dan instance (by Structured Dynamics). MyPeg.ca has been developed using the complete OSF (Open Semantic Framework) technologies stack. It is the reason why I (we) are really proud to start writing about this new innovative project. Mike also published an article that talk about other characteristics of the Peg project.

However, this project would not have been possible without the vision and the dedication of the IISD and the United Way of Winnipeg teams along with their partners. Also, it would not have been that well designed without Tactica‘s high quality graphics and design work.

MyPeg.ca’s Technology Stack

The project fully integrates, and leverages, the OSF (Open Semantic Framework) technologies stack and is based on the Citizen Dan community indicators principles. In the coming weeks, I will write about all and every aspects of the portal, however let’s take a first general overview of what is in the box.

The OSF stack is represented by this beautiful semantic muffin:

OSF layers

Everything starts with MyPeg’s existing assets:

1. Their Peg Framework which is the conceptual framework they created to analyze different facets of their community by leveraging a series of hundreds of indicators.
2. The indicators data that they aggregated and collected from different federal, provincial, municipal and local sources
3. The interviews they are performing with tens, and eventually hundreds, of Winnipeg citizens

Then all this data has been imported into the structWSF semantic data management framework by using two other pieces of technology:

1. The indicators data is described using the commON irON profile, and is maintained by the IISD team using a set of Excel spreadsheets. Then the dataset have been imported using the structImport conStruct module.
2. The interviews have been analyzed, tagged and imported in the system by using the Scones service and its structScones conStruct user interface.

Once all the data gets imported into the structWSF instance, it becomes available to all the conStruct modules, all the Semantic Components and all other tools that communicate with the structWSF web service endpoints.

Then ontologies have been used to describe the Peg Framework and to describe all the attributes of all the records (Neighborhoods, Cities, Community Areas and Stories). Already existing ontologies such as SCO have also been used for different criteria (such as driving the usage of the Semantic Component tools).

Then the sRelationBrowser, sDashboard, sMap, sStory, sBarChart and the sLinearChart Semantic Components along with the PortableControlApplication and Workbench applications have been used by Peg to create, manage, explore and publish information from their datasets.

Finally, the entire portal is published using Drupal and the set of conStruct modules. conStruct is the user interface to the structWSF web service endpoints. The mix of Drupal & conStruct templating technologies make it the perfect match to expose all the data, in different ways, by embedding different tools (such as the Semantic Components) depending on different criteria (user permissions, how the information is described into the system, etc.).

This is not a simple technology stack. However, this MyPeg.ca project is a good example of how an organization that never worked with semantic web technologies in the past have been able to has a long term vision of its objectives and how it understands that semantic technologies could help it to reach the aims of its vision. Then it demonstrates how everything has been integrated in an innovative Web portal.

Next Steps…

As I said above, in the coming weeks I will write about each of these technologies. I will show how each of them have been leveraged into the MyPeg.ca portal: how such generic tools have been used for highly specific tasks within the Peg project. Here is an overview of what is coming, where each main topic will result in a new blog post:

• How to integrate MyPeg indicators data into any Web application by using the structWSF web service endpoint
• Querying the MyPeg datasets, the geeky way, using the SPARQL endpoint
• Six ways to get data out of the system
  • By using the CrudRead/Search/Browse web service endpoints
  • By querying the SPARQL endpoint
  • By dereferencing record URIs
  • By using the export features on any record view pages
  • By using the export features of the search/browse modules pages
  • By using the structExport conStruct module

• How to create and maintain indicators dashboard sessions using the Workbench application
• How to integrate dashboard sessions into any Web page
  • Displaying dashboard sessions into record pages
  • Displaying dashboard sessions while using the Peg framework explorer tool
  • Displaying dashboard sessions into indicators analysis reports

• How to use maps:
  • as an information browsing tool
  • to contextualize instance records
  • to contextualize information in dashboard sessions

• How to use the explorer (sRelationBrowser) to browse conceptual structure and to display all kind of related information at each step
• Use of Scones to analyze, tag, index and display unstructured data
• Use of ontologies to drive the system
  • How ontologies are used to describe conceptual frameworks that drive these portals
  • How ontologies are used to drive the semantic components (SCO)

• Use of the commON irON profile and conStruct to serialize indicators data and to import it into the system
  • The benefits of commON as a common ground between the semantic web practitioner and the client.
  • commON as a wonderful format to manage indicator related datasets by indicators practitioners.

So stay tuned, because plenty of innovative stuff is coming!

One thing that was hard to do with structWSF was explaining what structWSF is, and how users can interact with it. For most people, structWSF was abstracted behind conStruct and they didn't know that each single functionalities of conStruct was bound to one, or multiple queries to one, or multiple, structWSF instance.

It is the reason why we took the time to write a complete structWSF interaction tutorial. This tutorial explains what the general structWSF architecture is, and it describes a series of general interaction usecases. We hope that this tutorial will helps developers and system implementators understanding the capabilities of structWSF and how they can use it.

You can read the complete structWSF Web Services Tutorial here.

Additionally, we released a new version of structWSF, conStruct and the irJSON Parser which are products of this toturial.

Two days ago we released irON: Instance Record and Object Notation (irON) Specification. irON is a new notation that has been created to describe instance records. irON records can be serialized in 3 different formats: irXML (XML), irJSON (JSON) and commON (CSV: mainly for spreadsheet manipulations).

The release of irON has already been covered at length on Mike’s blog and in Structure Dynamics’s press room; so I won’t talk more about it here.

irON Parsers

What I am happy to release today are the first two parsers that can be used to parse and validate irON datasets of instance records. The first two parsers that have been developed so far are the ones for irJSON and commON. Each parser has been developed in PHP and is available under the Apache 2 licence. Now, lets take a look at each of them

irJSON Parser

The irJSON parser package can be downloaded here. Additionally, the source code can be browsed here.

First of all, to understand the code, you have to understand the specification of the irJSON serialization.

The irON parser package is everything you need to test and use the parser. The package is composed of the following files:

• test.php – If you want to quick-start with this package, just run this test.php script and you will have an idea of what it can do for you. This script just runs the parser over a irJSON test file, and shows you some validation errors along with the internal parsed structure of the file. From there, you can simply use the irJSONParser class, with the structure that is returned to do whatever is needed for you: adding the information in you database, converting the data to another format, etc.
• irJSONParser.php – This is the irJSON parser class. It parses the irJSON file and populates its internal structure that is composed of instances of the classes below.
• Dataset.php – This class defines a Dataset records with all its attributes. It is the object that the developed has to manipulate that comes from the parser.
• InstanceRecord.php – This class defines an Instance Records with all its attributes. It is the object that the developed has to manipulate that comes from the parser.
• StructureSchema.php – This class defines a Structure Schema records with all its attributes. It is the object that the developed has to manipulate that comes from the parser.
• LinkageSchema.php – This class defines a Linkage Schema records with all its attributes. It is the object that the developed has to manipulate that comes from the parser.

The irJSON parser also validates the incoming irJSON files according to these three levels of validation:

1. JSON well-formedness validation – The first validation test occurs on the JSON serialization itself. A JSON file has to be a well formed in order to be processed. An error at this level will raise an error to the user.
2. irJSON well-formedness validation – Once JSON is parsed and well formed, the parser make sure that the file is irJSON well-formed. If it is not well formed according to the irJSON spec, an error will be raised to the user.
3. Structure Schema validation – The last validation that occurs is between instance records, and their related (if available) Structure Schema. If a validation error happens at this level, a notice will be raised to the user.

You can experiment with some of these validation errors and notices by running the test.php script in the package.

With this package, developers can already start to parse irJSON files and to integrate them with some of their prototype projects.

commON Parser

The commON parser package can be downloaded here. Additionally, the source code can be browsed here.

To understand the code, you have to understand the specification of the commON serialization.

The commON parser package is everything you need to test the parser. The package is composed of the following files:

• test.php – If you want to quick-start with this package, just run this test.php script and you will have an idea of what it can do for you. This script just run the parser over a file, and shows you some validation errors along with the internal parsed structure of the file. From there, you can simply use the CommonParser class, with the structure that is returned to do whatever is needed for you: adding the information in you database, converting the data to another format, etc.
• CommonParser.php – This is the commON parser class. It parses the commON file and populates its internal structure that is described in the code. the parser.

The commON parser also validates the incoming commON files according to these two levels:

1. CSV well-formedness validation – The first validation test occurs on the CSV serialization itself. A CSV file has to be a well formed in order to be processed. An error at this level will raise an error to the user.
2. commON well-formedness validation – Once CSV is parsed and well formed, the parser make sure that the file is CSV well-formed. If it is not well formed according to the CSV RFC, an error will be raised to the user.

You can experiment some of these validation errors and notices by running the test.php script in the package.

With this package, developers can already start to parsing commON files and to integrate them with some prototypes of their projects.

The commON parser is less advanced than the irJSON one. For example, the implementation of the “dataset” and the “schema” processor keywords are not yet done. Other keywords haven’t (yet) been integrated too. Take a look at the source code to know what is currently missing.

In any case, a lot of things can currently be done with this parser. We will publish specific commON usage use-cases in the coming weeks that will shows people are we are using commON internally and how we will expect our customers to use it to create and maintain different smaller datasets.

Conclusion

These are the first versions of the irJSON and commON parsers. We have to continue to development to make them perfectly reflecting the current and future irON specification. We yet have to write the irXML parser too.

I would encourage reporting any issues with these parsers, or any enhancement suggestions, on this issue tracked.

All discussions regarding these parsers and the irON specification document should happen on the irON group mailing list here.

Finally, another step for us will be to embed these parsers in converter web services for structWSF.