Open Semantic Framework: Major New Releases

Frederick Giasson
Structured Dynamics has just released new versions of the three main packages that compose the Open Semantic Framework: structWSF, conStruct and the Semantic Components. This is the end result of about 6 months of work that has been sponsored by multiple different projects.

A lot of new features have been added and some web-services/modules/components have been greatly updated. Let’s take a look at the major improvements.

structWSF

The main additions to structWSF are the new Scones, commON converter, and Tracker web service endpoints. Multiple bugs have also been fixed in different other endpoints.

Semantic Components

Multiple things happened related to the semantic components. First, the Workbench application, along with the Dashboard and the sStory components, have been released. These are the major new components that have been implemented in the recent MyPeg.ca public Winnipeg City indicators data system that was released last week.

Also, the sRelationBrowser and the sMap components got major enhancements as well.

conStruct

The main additions to conStruct were two new modules: structAppend and structScones. structAppend has been created to help people manage the datasets they import in their structWSF instances. You can read more about dataset management strategies using structAppend by reading this tutorial page and this other one. structScones is a user interface to the Scones web service endpoint. It takes unstructured text as input, tags concepts and named entities to that text, and then indexes the result in the structWSF instance. It also manages the creation of the files used by the sStory semantic component.

Finally the structCreate and structUpdate modules have been upgraded and are now using the Instance Records Forms Format. These forms will eventually be generated automatically by the next generation of the ontologies management structWSF web service endpoints.

Conclusion

All these new releases, enhancements and bug fixes are the fruits from the great cooperation Structured Dynamics has with its clients. All of these projects have been sponsored by different entities, and each of them has benefitted from what others helped to develop. We have been fortunate to have clients that also share an understanding of open source benefits.

Structured Dynamics’ open source strategy is working well. The end result is a better open source data ingest, management, analysis and publication via the Open Semantic Framework.

We are looking forward to continuing this progress via follow-ons with our current clients. And, of course, we are always looking for new opportunities with new clients.

MyPeg.ca – A Community Indicators Web Portal Using Semantic Web Technologies

Frederick Giasson
Now that the MyPeg.ca project has been unveiled at the Winnipeg Poverty Reduction Partnership Forum, I can now start to write about each and every feature of this innovative website. Peg

MyPeg.ca is a public indicators Web portal for the Canadian city of Winnipeg. It is supported by an open-source semantic web framework called OSF. This initial beta version of the Web portal emphasizes the integration, management, exploration and display of a few hundred Well-being indicators’ data for the city.

This community indicators portal is currently the best example of a Citizen Dan instance (by Structured Dynamics). MyPeg.ca has been developed using the complete OSF (Open Semantic Framework) technologies stack. It is the reason why I (we) are really proud to start writing about this new innovative project. Mike also published an article that talk about other characteristics of the Peg project.

However, this project would not have been possible without the vision and the dedication of the IISD and the United Way of Winnipeg teams along with their partners. Also, it would not have been that well designed without Tactica‘s high quality graphics and design work.

MyPeg.ca’s Technology Stack

The project fully integrates, and leverages, the OSF (Open Semantic Framework) technologies stack and is based on the Citizen Dan community indicators principles. In the coming weeks, I will write about all and every aspects of the portal, however let’s take a first general overview of what is in the box.

The OSF stack is represented by this beautiful semantic muffin:

OSF layers
OSF layers

Everything starts with MyPeg’s existing assets:

  1. Their Peg Framework which is the conceptual framework they created to analyze different facets of their community by leveraging a series of hundreds of indicators.
  2. The indicators data that they aggregated and collected from different federal, provincial, municipal and local sources
  3. The interviews they are performing with tens, and eventually hundreds, of Winnipeg citizens

Then all this data has been imported into the structWSF semantic data management framework by using two other pieces of technology:

  1. The indicators data is described using the commON irON profile, and is maintained by the IISD team using a set of Excel spreadsheets. Then the dataset have been imported using the structImport conStruct module.
  2. The interviews have been analyzed, tagged and imported in the system by using the Scones service and its structScones conStruct user interface.

Once all the data gets imported into the structWSF instance, it becomes available to all the conStruct modules, all the Semantic Components and all other tools that communicate with the structWSF web service endpoints.

Then ontologies have been used to describe the Peg Framework and to describe all the attributes of all the records (Neighborhoods, Cities, Community Areas and Stories). Already existing ontologies such as SCO have also been used for different criteria (such as driving the usage of the Semantic Component tools).

Then the sRelationBrowser, sDashboard, sMap, sStory, sBarChart and the sLinearChart Semantic Components along with the PortableControlApplication and Workbench applications have been used by Peg to create, manage, explore and publish information from their datasets.

Finally, the entire portal is published using Drupal and the set of conStruct modules. conStruct is the user interface to the structWSF web service endpoints. The mix of Drupal & conStruct templating technologies make it the perfect match to expose all the data, in different ways, by embedding different tools (such as the Semantic Components) depending on different criteria (user permissions, how the information is described into the system, etc.).

This is not a simple technology stack. However, this MyPeg.ca project is a good example of how an organization that never worked with semantic web technologies in the past have been able to has a long term vision of its objectives and how it understands that semantic technologies could help it to reach the aims of its vision. Then it demonstrates how everything has been integrated in an innovative Web portal.

Next Steps…

As I said above, in the coming weeks I will write about each of these technologies. I will show how each of them have been leveraged into the MyPeg.ca portal: how such generic tools have been used for highly specific tasks within the Peg project. Here is an overview of what is coming, where each main topic will result in a new blog post:

  • How to integrate MyPeg indicators data into any Web application by using the structWSF web service endpoint
  • Querying the MyPeg datasets, the geeky way, using the SPARQL endpoint
  • Six ways to get data out of the system
    • By using the CrudRead/Search/Browse web service endpoints
    • By querying the SPARQL endpoint
    • By dereferencing record URIs
    • By using the export features on any record view pages
    • By using the export features of the search/browse modules pages
    • By using the structExport conStruct module
  • How to use the explorer (sRelationBrowser) to browse conceptual structure and to display all kind of related information at each step
  • Use of Scones to analyze, tag, index and display unstructured data
  • Use of ontologies to drive the system
    • How ontologies are used to describe conceptual frameworks that drive these portals
    • How ontologies are used to drive the semantic components (SCO)
  • Use of the commON irON profile and conStruct to serialize indicators data and to import it into the system
    • The benefits of commON as a common ground between the semantic web practitioner and the client.
    • commON as a wonderful format to manage indicator related datasets by indicators practitioners.

So stay tuned, because plenty of innovative stuff is coming!

Semantic Components : the Genesis

Frederick Giasson

Everything started with this prototype: structSearch. The goal was to figure out if Flex, and some of its open source libraries, were about to make a good data visualization framework for Structured Dynamics. As it turned out, it was making more than a good data visualization framework, as the Semantic Component Library is now one of the central libraries maintained by Structured Dynamics.

It was about one year ago.

But it is more than a demo, it is more than just a new data visualization framework. It is a set of technologies, it is a process by which you can do data integration, management, publication and visualization faster, better and for a fraction of the price. It is what we do at Structured Dynamics, and we hope continuing to share our findings to help communities to move faster, better and for the best.

What we do here is to demonstrate one of the facet of this new architecture.

strutSearch

The structSearch demo is interesting because it is a good example of what we call an Ontology Driven Application. Basically, the user interface of the application behave differently depending of the data that it gets as input.

In the example below, the structSearch tool gets all its data from a call to a structWSF instance where the Semantic Sweet Tools List dataset is hosted. You can try a search with the acronym “rdf“:

[kml_flashembed publishmethod=”static” fversion=”10.0.0″ movie=”http://fgiasson.com/flex/demos/structSearch/structSearch.swf” width=”500″ height=”700″ targetclass=”flashmovie” fvars=”baseAppUrl=http://fgiasson.com/flex/demos/structSearch/”]Get Adobe Flash player

[/kml_flashembed]

Note: if you have issues loading this Flash movie, it may be because the feed reader you are using, and what it does with some of the code within this blog post. I would suggest you to go to the website directly to experiment this tool.

How it works

The general workflow is somewhat simple; it is composed of only three parts:

1.����� A Resultset (which is composed of one or multiple records descriptions)

2.����� An application layout (a MXML file)

3.����� And a rendered user interface

These three steps are shown in this image (click to enlarge):

Semantic Component Layout
Semantic Component Layout

The complete process is discussed on the TechWiki. The structSearch demo leverages one of the semantic components: the sGenericBox. This box is a generic record infobox: it tries to display all the information related to a record�s description that is not handled by any other part of the application layout layer. So, most of the attributes/values are handled by the application, but for all the remaining, unhandled, attribute/values, they get listed in the generic box.

What is interesting with this three steps workflow that the users starts with one thing: a set of attributes/values describing a record. Then, they build a user interface that binds different attributes to controls that will be used to display their values. Finally, depending on a record’s description, and the application layout, the user interface generated will looks like/behave differently.

Why?

There is no magic in there, and the idea of adaptive user interfaces is certainly not new. However, what I think that people should keep in mind after reading this blog post is that the this specific workflow, along with these specific technologies helps the: integration, management, publication and visualization of different data sources. I hear you saying: yes and what? There are hundred of systems that tries to do this. Yes, that is true. But I would answer to that question with another question: yes, but at what price? By this I mean: can you develop as quickly and effectively as you could with these new methods, processes and technologies?

At least, this is what we figured out.

fvars

Christmas is not there yet!

Frederick Giasson
Christmas is coming: it is November next week; all leaves have fallen; big snowfalls are coming… but Christmas is not there yet! We have plenty of things to do before being able to settle down, for a few days, for Christmas.It has been some time since we last released new code related to OSF (Open Semantic Framework). It has also been some time since we last released new demos too.

However, this year, gifts will be delivered before Christmas. We are planning to release a multiple of things in the next two months, and I will personally restart blogging more often about this stuff.

structWSF & construct

The last release versions of structWSF and conStruct have been released in June. Since then, we updated some code in the dev branches of the SVNs, but plenty of code yet has to be released for conStruct. Expect a new official release of both products in November.

UMBEL

As mentioned in this press release, we will release two brand new versions of UMBEL (v080 and then v1) before the end of December. These will be major releases of the UMBEL vocabulary and its related reference concepts structures. There will also be other new goodies that I cannot mention here as I write.

Semantic Component Library

The Semantic Component Library is the product that changed the most in the last few months. It got heavily updated because of its usage in different projects. Some new semantic components will be released, but more importantly, existing components get heavily updated as well with new features, bug fixes, etc.

Demos

Some of our clients should release demo websites before Christmas. Most of them heavily leverage the Semantic Components that got integrated in conStruct and that are interacting with structWSF instances to get the proper data to display.

These will give some good examples on how these different tools can be used, and how heavily they can be skinned to properly get integrated in different kind of Web portals.

I will naturally write about these, when the come available, on this blog.

Documentation

We are updating the TechWiki documentation weekly. However, expect more new developer related documentation before Christmas. More precisely, we will write more documentation that explains the internal architecture and minding with structWSF and conStruct.