Getting Data Out of MyPeg.ca using structWSF Endpoints

Frederick Giasson
A few weeks ago I presented the new MyPeg.ca community indicators web portal for Winnipeg’s citizens. I explained how in MyPeg.ca we leverage Structured Dynamics’ semantic technologies stack (akaThe Semantic Muffin). Today’s blog post explains one facet of the project that shows how external agents (people, services, software, etc.) can interact with the system’s indicator datasets using the structWSF web service endpoints.Since this post focuses only on data export, I suggest you read the structWSF Web Services Tutorial for a complete overview of how the endpoints architecture works.

Merging Pipes

Two Main structWSF Characteristics: Accessibility & Management

structWSF is a set of 22 web service endpoints that lets you integrate data from different sources, manage that integrated data, and publish it via different communication channels such as web pages, software applications, etc.

Obviously, the main characteristic of this framework is that everything is a web service. This means that all functionality of the system can be accessed from anywhere on the Internet. However, this doesn’t mean that everything is open like a snack-bar. In fact, there are two levels of accessibility: (1) access to the web service endpoint’s URL, and (2) access to the content of datasets hosted on structWSF. Depending on the usecase, people could restrict the direct access to the web service endpoint(s) by properly configuring their web server, others could choose to let anyone access the endpoints, but would restrict the access to the dataset(s) hosted by structWSF. In case of MyPeg.ca, the sponsor chose to open the access to their web service endpoints and datasets.

Just by surfing on the MyPeg.ca portal, you are already leveraging these endpoints in multiple different ways. First, each time you generate a browse or a search Web page, you are telling the web server to send multiple queries to different endpoints; then the web page’s content will be populated with that information and presented to you. But, each time you click on an explorer node, your web browser is also sending queries to exactly the same web service endpoints. So, in one case a PHP script acts to query the endpoints; and, in other cases, a Flash Semantic Component does. Depending, all structWSF data can be accessed from quite different environments.

The other main characteristic of structWSF is that any kind of data can be imported in, and exported out, of the system. structWSF leverages RDF (Resource Description Framework) as the canonical data format that can be used to express any other formats. It is because of the usage of RDF that structWSF can act as an effective ETL (Extract, transform, load) system. Depending on the web service endpoint, the output formats currently supported by most of the endpoints are:

But the architecture of the web service endpoints can easily accommodate other formats if needed for a specific usecase.

Getting Data Out Of MyPeg.ca

Now, how can you get data out of MyPeg.ca? There are really two methods. This blog post discusses the CRUD: Read, Browse and Search web service endpoints. In my next blog post, I will focus on using the SPARQL web service endpoint to do the same.

All of the query examples in this blog post will use a tool called Curl to send the queries and to get back the resultsets. I encourage you to download and use that tool to test these endpoints and to gain a feeling for how it works. Also note that only the first record of each resultset is recorded below (of course, the actual results include all records).

Browse

The Browse web service endpoint is used to return lists of records. These records can also be filtered according to their provenance (dataset), type and the attributes that describe them. Now, let’s see how you can use this web service to get data out of MyPeg.ca.

First, there are three datasets available to the public:

  1. Well-being Indicators (http://www.mypeg.ca/wsf/datasets/258/)
  2. Stories (http://www.mypeg.ca/wsf/datasets/272/)
  3. PEG Framework (http://www.mypeg.ca/wsf/datasets/249/)

The resultsets can be serialized using one of these four different formats:

  • text/xml (structXML)
  • application/json (structXML in JSON)
  • application/rdf+xml (RDF/XML)
  • application/rdf+n3 (RDF/N3)

Note: if one of your desired formats is not directly available at the endpoint level, you can always use one of the converter web service endpoints such as: commON, irJSON or TSV/CSV.

Get the first 10 results of the Stories dataset in structXML

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: text/xml” “http://www.mypeg.ca/ws/browse/” -d “attributes=all&types=all&datasets=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2F&items=10&page=0&inference=on&include_aggregates=true”[/cc]

StructXML resultset:

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]

<?xml version=”1.0″ encoding=”utf-8″?>
<resultset>
<prefix entity=”owl” uri=”http://www.w3.org/2002/07/owl#”/>
<prefix entity=”rdf” uri=”http://www.w3.org/1999/02/22-rdf-syntax-ns#”/>
<prefix entity=”rdfs” uri=”http://www.w3.org/2000/01/rdf-schema#”/>
<prefix entity=”wsf” uri=”http://purl.org/ontology/wsf#”/>
<subject type=”http://purl.org/ontology/muni#Story” uri=”http://www.mypeg.ca/wsf/datasets/272/resource/AgeOpportunity”>
<predicate type=”http://purl.org/dc/terms/isPartOf”>
<object type=”http://rdfs.org/ns/void#Dataset” uri=”http://www.mypeg.ca/wsf/datasets/272/”/>
</predicate>
<predicate type=”http://purl.org/ontology/iron#prefLabel”>
<object type=”rdfs:Literal”>Age &amp; Opportunity</object>
</predicate>
<predicate type=”http://purl.org/dc/terms/created”>
<object type=”rdfs:Literal”>2010-10-28T19:38:58+00:00</object>
</predicate>
<predicate type=”http://purl.org/ontology/bibo/abstract”>
<object type=”rdfs:Literal”>Amanda Macrae, Deborah Lorteau and Stacey Miller work for Age and Opportunity.
The majority of clients are older adults living at lower socio economic status. When addressing the housing issue they say, “In a nutshell, it’s dire.” There is simply not enou…</object>
</predicate>
<predicate type=”http://purl.org/ontology/peg#interviewee”>
<object type=”rdfs:Literal”>Amanda Macrae, Deborah Lorteau, Stacey Miller</object>
</predicate>
<predicate type=”http://purl.org/ontology/peg#interviewer”>
<object type=”rdfs:Literal”>Molly Johnson</object>
</predicate>
<predicate type=”http://purl.org/ontology/peg#storyRelatedAgencyProgram”>
<object type=”rdfs:Literal”>Age &amp; Opportunity</object>
</predicate>
<predicate type=”http://purl.org/ontology/sco#storyAnnotatedTextUri”>
<object>http://www.mypeg.ca/scones/AgeOpportunity.xml</object>
</predicate>
<predicate type=”http://purl.org/ontology/sco#storyTextUri”>
<object type=”rdfs:Literal”>http://www.mypeg.ca/scones/AgeOpportunity.txt</object>
</predicate>
</subject>
</resultset>

[/cc]

Get the 10 first results from all datasets that are records of type Neighborhoods in RDF/XML

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/rdf+xml ” “http://www.mypeg.ca/ws/browse/” -d “attributes=all& type=http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23Neighborhood &datasets=all&items=10&page=0&inference=on&include_aggregates=true”[/cc]

RDF/XML resultset:

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]

<?xml version=”1.0″?>
<rdf:RDF  xmlns:owl=”http://www.w3.org/2002/07/owl#” xmlns:rdf=”http://www.w3.org/1999/02/22-rdf-syntax-ns#” xmlns:rdfs=”http://www.w3.org/2000/01/rdf-schema#” xmlns:wsf=”http://purl.org/ontology/wsf#” xmlns:ns0=”http://purl.org/ontology/peg#” xmlns:ns1=”http://purl.org/dc/terms/” xmlns:ns2=”http://purl.org/ontology/iron#” xmlns:ns3=”” xmlns:ns4=”http://purl.org/dc/elements/1.1/” xmlns:ns5=”http://purl.org/ontology/aggregate#”>

<ns0:Component rdf:about=”http://purl.org/ontology/peg/framework#Safety”>
<ns1:isPartOf rdf:resource=”http://www.mypeg.ca/wsf/datasets/249/” />
<ns2:prefLabel>Safety</ns2:prefLabel>
<ns2:altLabel>safety</ns2:altLabel>
<ns3:>safety</ns3:>
<ns4:description>Safety is the state of being “safe”, the condition of being protected against physical, social, spiritual, financial, political, emotional, occupational, psychological, educational or other types or consequences of failure, damage, error, accidents, harm or any other event which could be considered non-desirable.</ns4:description>
<rdfs:comment>Includes the idea of safety prevention</rdfs:comment>
<rdfs:seeAlso>http://en.wikipedia.org/wiki/Safety</rdfs:seeAlso>
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#HouseholdIncome” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#LowIncomeCutOffAfterTax” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#MarketBasketMeasure” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#ParticipationInSportsAndRecreation” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#MaternalSocialIsolation” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#PersonalSafety” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#EarlyDevelopmentInstrument” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#HighSchoolGraduationRate” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#LongTermUnemployment” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#TeenageBirths” />
<ns0:isComponentOf rdf:resource=”http://purl.org/ontology/peg/framework#BasicNeeds” />
<ns0:isComponentOf rdf:resource=”http://purl.org/ontology/peg/framework#Poverty” />
</ns0:Component>
</rdf:RDF>

[/cc]

Search

The Search web service endpoint is also used to return lists of records. These records should match a search string and can also be filtered according to their provenance (dataset), type and the attributes that describe them.

The same mime types and datasets as the ones for the Browse web service are available for the Search endpoint.

Searching for records with the keyword “poverty” and get resultsets in RDF/N3

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/rdf+n3” “http://www.mypeg.ca/ws/search/” -d “query=poverty&datasets=all&items=10&page=0&inference=on&include_aggregates=true”[/cc]

RDF/N3 resultset:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix wsf: <http://purl.org/ontology/wsf#> .

<http://purl.org/ontology/peg/framework#Poverty> a <http://purl.org/ontology/peg#CrossCuttingIssue> ;
<http://purl.org/dc/terms/isPartOf> <http://www.mypeg.ca/wsf/datasets/249/> ;
<http://purl.org/ontology/iron#prefLabel> “””Poverty””” ;
<http://purl.org/dc/elements/1.1/description> “””Poverty is not having the sufficient resources, capabilities, choices, security and power necessary to enjoy an adequate standard of living.  Poverty includes much more than a lack of money.  It includes being excluded from ordinary living patterns, customs and activities.  Consequently, people living in poverty are often unable to participate fully in their communities or to reach their full potential.””” ;
<http://www.w3.org/2000/01/rdf-schema#seeAlso> “””http://en.wikipedia.org/wiki/Poverty””” .

[/cc]

CRUD: Read

The Browse and Search web service endpoints are really used to find lists of records according to some provided criteria. However, the complete description of these records is not returned by these endpoints, but only the information necessary to create the proper list to display to users in a user interface. So, to get the complete description of a record (or multiples thereof), you have to use the CRUD: Read web service endpoint. Also, sometimes you may get a reference to a record hosted on a structWSF node, then CRUD: Read is the way to get its full description.

Get the full description of the Ida story in irJSON

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/iron+json” “http://www.mypeg.ca/ws/crud/read/?uri=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2Fresource%2FIda&dataset=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2F&include_reification=true&include_linksback=false[/cc]

irJSON resulset:

[cc lang=’javascript’ line_numbers=’false’ escaped=’true’]

{
“dataset”: {
“linkage”: [
{
“linkedType”: “application/rdf+xml”,
“attributeList”: {
“created”: {
“mapTo”: “http://purl.org/dc/terms/created”
},
“isAbout”: {
“mapTo”: “http://umbel.org/umbel#isAbout”
},
“prefLabel”: {
“mapTo”: “http://purl.org/ontology/iron#prefLabel”
},
“interviewee”: {
“mapTo”: “http://purl.org/ontology/peg#interviewee”
},
“interviewer”: {
“mapTo”: “http://purl.org/ontology/peg#interviewer”
},
“abstract”: {
“mapTo”: “http://purl.org/ontology/bibo/abstract”
},
“storyVideoAudio”: {
“mapTo”: “http://purl.org/ontology/peg#storyVideoAudio”
},
“storyAnnotatedTextUri”: {
“mapTo”: “http://purl.org/ontology/sco#storyAnnotatedTextUri”
},
“storyTextUri”: {
“mapTo”: “http://purl.org/ontology/sco#storyTextUri”
}
},
“typeList”: {
“Story”: {
“mapTo”: “http://purl.org/ontology/muni#Story”
}
}
}
]},
“recordList”: [
{
“id”: “http://www.mypeg.ca/wsf/datasets/272/resource/Ida”,
“type”: “Story”,
“created”: “2010-10-28T18:11:27+00:00”,
“isAbout”: [
{
“ref”: “@@http://purl.org/ontology/peg/framework#EducationAndLearning”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Health”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Program”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Income”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Poverty”
}     ],
“prefLabel”: “Ida”,
“interviewee”: “Ida”,
“interviewer”: “Christa Rust”,
“abstract”: “‘Poverty is earning just enough to get by; never having money for extras.’\n\nIda is the mother of two grown children, eight years apart.  She lives in a small bachelor suite, which costs her $428 per month, or 62% of her income.  She volunteers twice a we…”,
“storyVideoAudio”: “http://www.youtube.com/v/0zIqtYhiHfM”,
“storyAnnotatedTextUri”: “http://www.mypeg.ca/scones/Ida.xml”,
“storyTextUri”: “http://www.mypeg.ca/scones/Ida.txt”
}
]
}

[/cc]

Get Well-Being record description with linkbacks in RDF+N3

The characteristic of this query is that I enabled the “include_linksback” parameter. This returns a reference to all the records, in the datasets hosted on the structWSF node, that refers to that target record.

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/rdf+n3” “http://www.mypeg.ca/ws/crud/read/?uri=http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing&datast=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F249%2F&registered_ip=self%3A%3A0&include_reification=true&include_linksback=true”[/cc]

RDF+N3 resultset:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

<http://purl.org/ontology/peg/framework#WellBeing> a <http://purl.org/ontology/peg/framework#WellBeing> ;
<http://purl.org/ontology/iron#prefLabel> “””Well-being””” ;
<http://purl.org/dc/elements/1.1/description> “””Well-being refers to the general quality of life experienced by individuals and communities. The elements of wellbeing include: the ability to meet basic needs, the economy, health, the built environment, governance, education and learning, the natural environment, and social vitality.””” ;
<http://purl.org/ontology/sco#displayComponent> <http://purl.org/ontology/sco#sRelationBrowser> .

<http://purl.org/ontology/peg/framework#WellBeing> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Economy> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Governance> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#BuiltEnvironment> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#NaturalEnvironment> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#SocialVitality> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#BasicNeeds> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#EducationAndLearning> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Health> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

[/cc]

General Endpoint Parameters

The general parameters available for each of these web services is provided in their respective TechWiki documentation. For that detailed information, see the Browse, Search, or CRUD: Read articles.

Conclusion

As you can see, agents can get different kinds of data from the MyPeg.ca portal by querying a set of web service endpoints. This is one way to get data out of the system. These data can then be accessed for indexing in other systems, for direct use, or for dynamic applications like browsing the nodes in the explorer.

This is one of the ways to get data out of the system. A user can also export that same information from the Export features on the Browse, Search and Record View pages. Also, other methods will be explained in the next blog posts from this MyPeg.ca series.

All in all, this shows how effective structWSF can be to integrate, manage and publish a wide range of data in different data formats. It also shows how completely different parts of your software architecture can leverage your information, the way you want, from anywhere on the Internet.

UMBEL Blooms with New Colors

Frederick Giasson
We are happy to announce the new, intermediary, UMBEL version 0.80. This is a major upgrade of the UMBEL ontology: both its vocabulary and its reference structure have been greatly enhanced, an upper structure called the SuperTypes has been added and everything got updated to OWL 2. You can read more about the overall changes on Mike’s blog post.

In this blog post I will focus on two topics: using some existing tools and frameworks to view and manage the reference concepts structure, and how one can use and leverage the coherency of the reference structure.

Navigating and Updating the Reference Structure

One thing that was lacking with the previous version of UMBEL was to have access to a user interface tool that would let you navigate and update the reference structure as you want. Because of the way the conceptual structure was created, it was hard for tools such as Protégé to load it because of all the individuals that were created (such as the SemSet individuals, etc.).

As stated in Mike’s blog post, we made significant changes to the UMBEL vocabulary, and how we instantiate the reference structure. Along with the OWL 2 upgrade, we made sure that the Protégé version 4.1 and the latest version of the OWLAPI could easily load both the UMBEL vocabulary and the reference structure.

Reasoning

One of the major additions to UMBEL v080 is the SuperTypes upper structure, an organizational layer above the UMBEL reference structure. We created these SuperTypes because we found that we could effectively cluster most UMBEL reference concepts into a small set of mostly distinct upper concepts (33 in fact, 29 of which are designed as disjoint).

This new SuperTypes structure helps us mine external sources of information by leveraging related concepts in the reference structure. Moreover, SuperTypes also help us perform easier, simpler, better and faster reasoning over the entire 21 K reference concepts structure.

Thus, SuperTypes provide a new tool to help determine if the UMBEL reference structure is consistent and coherent within itself. This is important, of course, to ensure that linkages between UMBEL and external ontologies is consistent and coherent as well.

So far, the entire reference concepts structure has been tested for its coherency according to the restrictions we defined at the level of the SuperTypes upper structure. Using different reasoners such as Pellet, Fact++ and Hermit (available by default with Protégé 4.1), we made sure that all the statements made between all the RefConcept classes and individuals, and all the statements made between these and the SuperTypes upper structure, are consistent within themselves. This method enabled us to find and fix some early assignment issues.

This new upper structure, along with its now consistent reference structure, helps provide confidence that statements based on UMBEL reference concepts are also consistent. And, all of this is made more testable by virtue of being able to use the OWL API and Protégé with its embedded reasoners.

How is Coherency Tested?

This is the core question. In fact, the more informative answer to this question will be part of a forthcoming blog post. But let’s start here.

The current way to check if the structure is coherent is by making sure that we don’t have an individual that belongs to two different SuperTypes that are stated to be disjoint. What we did with the SuperType upper structure is really simple: we categorized each and every RefConcept (using rdfs:subClassOf) under a SuperType. Most of the SuperTypes are disjoint: this means that if an individual is of rdf:type for two SuperTypes that are stated to be disjoint, then you will end-up with an incoherent structure because you are making a statement that is not permitted by the reference structure.

So, the way to check if your statements are coherent according to this structure, is to make your statements (right now, in terms of individual instantiation), and then to check using a reasoner such as Pellet. There is now a general testing structure to see if any ontology is coherent with respect to the UMBEL reference structure.

In the next blog post in this series, I will tell you how to use exactly the same method for coherency testing, but now for testing if linkages between external ontologies and the UMBEL reference structure are consistent. In that case, you will make the class-to-class assertions you want, and then you will instantiate individuals of these classes, then run the reasoner. Then, the reasoner will tell you if your ontology is still consistent according to the structure and the new statements you created.

Next Step

In parallel with these tutorials, we are also working hard on the next version of UMBEL. As outlined in the Next Changes section of the new UMBEL website, the next step is to release UMBEL v1.0, with a set of new features, before Christmas.

Building conStruct Templates

Frederick Giasson
The templating engine in conStructis one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

  • Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.
  • Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.
  • Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on “text”, you will get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.


As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

 

Building conStruct Templates

The templating engine in conStruct is one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…

Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

�Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.

�Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.

�Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on �text�, you will

Building conStruct Templates

The templating engine in conStruct is one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…

Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

  • Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.
  • Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.
  • Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on �text�, you will get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.

As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.

As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

Open Semantic Framework: Major New Releases

Frederick Giasson
Structured Dynamics has just released new versions of the three main packages that compose the Open Semantic Framework: structWSF, conStruct and the Semantic Components. This is the end result of about 6 months of work that has been sponsored by multiple different projects.

A lot of new features have been added and some web-services/modules/components have been greatly updated. Let’s take a look at the major improvements.

structWSF

The main additions to structWSF are the new Scones, commON converter, and Tracker web service endpoints. Multiple bugs have also been fixed in different other endpoints.

Semantic Components

Multiple things happened related to the semantic components. First, the Workbench application, along with the Dashboard and the sStory components, have been released. These are the major new components that have been implemented in the recent MyPeg.ca public Winnipeg City indicators data system that was released last week.

Also, the sRelationBrowser and the sMap components got major enhancements as well.

conStruct

The main additions to conStruct were two new modules: structAppend and structScones. structAppend has been created to help people manage the datasets they import in their structWSF instances. You can read more about dataset management strategies using structAppend by reading this tutorial page and this other one. structScones is a user interface to the Scones web service endpoint. It takes unstructured text as input, tags concepts and named entities to that text, and then indexes the result in the structWSF instance. It also manages the creation of the files used by the sStory semantic component.

Finally the structCreate and structUpdate modules have been upgraded and are now using the Instance Records Forms Format. These forms will eventually be generated automatically by the next generation of the ontologies management structWSF web service endpoints.

Conclusion

All these new releases, enhancements and bug fixes are the fruits from the great cooperation Structured Dynamics has with its clients. All of these projects have been sponsored by different entities, and each of them has benefitted from what others helped to develop. We have been fortunate to have clients that also share an understanding of open source benefits.

Structured Dynamics’ open source strategy is working well. The end result is a better open source data ingest, management, analysis and publication via the Open Semantic Framework.

We are looking forward to continuing this progress via follow-ons with our current clients. And, of course, we are always looking for new opportunities with new clients.

MyPeg.ca – A Community Indicators Web Portal Using Semantic Web Technologies

Frederick Giasson
Now that the MyPeg.ca project has been unveiled at the Winnipeg Poverty Reduction Partnership Forum, I can now start to write about each and every feature of this innovative website. Peg

MyPeg.ca is a public indicators Web portal for the Canadian city of Winnipeg. It is supported by an open-source semantic web framework called OSF. This initial beta version of the Web portal emphasizes the integration, management, exploration and display of a few hundred Well-being indicators’ data for the city.

This community indicators portal is currently the best example of a Citizen Dan instance (by Structured Dynamics). MyPeg.ca has been developed using the complete OSF (Open Semantic Framework) technologies stack. It is the reason why I (we) are really proud to start writing about this new innovative project. Mike also published an article that talk about other characteristics of the Peg project.

However, this project would not have been possible without the vision and the dedication of the IISD and the United Way of Winnipeg teams along with their partners. Also, it would not have been that well designed without Tactica‘s high quality graphics and design work.

MyPeg.ca’s Technology Stack

The project fully integrates, and leverages, the OSF (Open Semantic Framework) technologies stack and is based on the Citizen Dan community indicators principles. In the coming weeks, I will write about all and every aspects of the portal, however let’s take a first general overview of what is in the box.

The OSF stack is represented by this beautiful semantic muffin:

OSF layers
OSF layers

Everything starts with MyPeg’s existing assets:

  1. Their Peg Framework which is the conceptual framework they created to analyze different facets of their community by leveraging a series of hundreds of indicators.
  2. The indicators data that they aggregated and collected from different federal, provincial, municipal and local sources
  3. The interviews they are performing with tens, and eventually hundreds, of Winnipeg citizens

Then all this data has been imported into the structWSF semantic data management framework by using two other pieces of technology:

  1. The indicators data is described using the commON irON profile, and is maintained by the IISD team using a set of Excel spreadsheets. Then the dataset have been imported using the structImport conStruct module.
  2. The interviews have been analyzed, tagged and imported in the system by using the Scones service and its structScones conStruct user interface.

Once all the data gets imported into the structWSF instance, it becomes available to all the conStruct modules, all the Semantic Components and all other tools that communicate with the structWSF web service endpoints.

Then ontologies have been used to describe the Peg Framework and to describe all the attributes of all the records (Neighborhoods, Cities, Community Areas and Stories). Already existing ontologies such as SCO have also been used for different criteria (such as driving the usage of the Semantic Component tools).

Then the sRelationBrowser, sDashboard, sMap, sStory, sBarChart and the sLinearChart Semantic Components along with the PortableControlApplication and Workbench applications have been used by Peg to create, manage, explore and publish information from their datasets.

Finally, the entire portal is published using Drupal and the set of conStruct modules. conStruct is the user interface to the structWSF web service endpoints. The mix of Drupal & conStruct templating technologies make it the perfect match to expose all the data, in different ways, by embedding different tools (such as the Semantic Components) depending on different criteria (user permissions, how the information is described into the system, etc.).

This is not a simple technology stack. However, this MyPeg.ca project is a good example of how an organization that never worked with semantic web technologies in the past have been able to has a long term vision of its objectives and how it understands that semantic technologies could help it to reach the aims of its vision. Then it demonstrates how everything has been integrated in an innovative Web portal.

Next Steps…

As I said above, in the coming weeks I will write about each of these technologies. I will show how each of them have been leveraged into the MyPeg.ca portal: how such generic tools have been used for highly specific tasks within the Peg project. Here is an overview of what is coming, where each main topic will result in a new blog post:

  • How to integrate MyPeg indicators data into any Web application by using the structWSF web service endpoint
  • Querying the MyPeg datasets, the geeky way, using the SPARQL endpoint
  • Six ways to get data out of the system
    • By using the CrudRead/Search/Browse web service endpoints
    • By querying the SPARQL endpoint
    • By dereferencing record URIs
    • By using the export features on any record view pages
    • By using the export features of the search/browse modules pages
    • By using the structExport conStruct module
  • How to use the explorer (sRelationBrowser) to browse conceptual structure and to display all kind of related information at each step
  • Use of Scones to analyze, tag, index and display unstructured data
  • Use of ontologies to drive the system
    • How ontologies are used to describe conceptual frameworks that drive these portals
    • How ontologies are used to drive the semantic components (SCO)
  • Use of the commON irON profile and conStruct to serialize indicators data and to import it into the system
    • The benefits of commON as a common ground between the semantic web practitioner and the client.
    • commON as a wonderful format to manage indicator related datasets by indicators practitioners.

So stay tuned, because plenty of innovative stuff is coming!