Unnatural Open Source

Frederick Giasson
I have never been an open source software advocate. In fact, like most people, I always wondered how companies could find a business advantage in developing open source softwares and how they could make money out of it to grow. It is nice to have open source softwares, but it is hard to imagine how you could justify putting thousands of hours in open source software projects if it is not only by passion.

In this post I will explain what I think is the main factor that put people, businesses and organizations on guard when come the time to think about open source softwares. In fact, I think it has much more to do with our nature: how we naturally are as human being, and much less to do with any real business related factors.

In a follow-up blog post, I will explain how Structured Dynamics embraced open source software, how we developed the company around the concept, and how we are managing the development of our project such that it benefits all our clients along with the company. But first, let’s try to figure out why much people are suspicious regarding open source softwares.

The Fear

“I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.”

– Dune, Frank Herbert

Have you ever heard someone telling you:

I found an incredible business idea! I am pretty sure that I am the first one to think about that. I will get some good money down the road!

Then, you naturally asked for more information about this great idea! And then the answer you got was something like:

Hooo! But I can’t tell you, this is really secret right now, at least until everything is ready to go.

Does this sound familiar? I does to me. I hear it often. But, why does people react that way? It is simply by fear: fearing that someone “steal” their ideas, start a company based on them, build projects or services that implemented them, and get rich while you are flipping burgers at McDonald’s.

To me, this is the main reason why people, organizations and businesses are suspicious regarding open source software: because of fear; fear of loosing something they don’t even have.

But the question is: is that rational? From my experience, and my understanding of how things works, I can certainly say that it is not. This way of thinking is not rational because it doesn’t take into account a few things:

  • The ability of others to do something with your ideas
  • The ability of others to have the vision you have for your ideas
  • The willingness of others to spend all their time and energy to make these ideas working
  • [raw]People tend to do what they want to do, and not what others wants­[/raw]

The same behavior seems to happen with open source projects. When I am explaining to people what we are doing, one of the first reaction is: why your work is open and free? Don’t you fear that someone steal your project and ideas? How can you make money if it is free, people will just run with it for themselves no?

The simple answer to all these question is: no. No we don’t feature that anybody steal our projects and ideas just by cloning them from the source control. We don’t because of the four reasons listed above. We don’t because we trust our vision and our abilities to implement it in our various open source projects. And yes we can sustain the company pretty well with these projects and it is what I will cover in my following blog post.

Conclusion

Non-Open Source softwares are just like when someone has a business idea “for the next big thing” and that doesn’t want to share it with anybody else because he think that someone will take that idea and run with it by himself. In fact, it is quite the opposite. I learned with experience that there is only one person (or organization) that can make such a great idea a relative success: the person (or organization) that lives for that idea. An idea is just an idea, and has nothing great in it, until it gets implemented, until the idea lives by itself, propelled by it most dedicated advocate: its creators and their boundless enthusiasm. Any idea would fail without this… and would worth nothing; it would just be an idea.

Volkswagen’s Use of structWSF in their Semantic Web Platform

Frederick Giasson

TribalDDB London, Volkswagen UK‘s partner, mentioned earlier this week that Volkswagen are using some parts of the Open Semantic Framework to develop the next generation of their online platform.

This story has been published by Jennifer Zaino’s in her article: Volkswagen: Das Auto Company is Das Semantic Web Company!

I can now talk about this project that uses some pieces of the framework that we have been developing for more than 3 years now.

The Objective

Volkswagen’s main objective behind the development of the next version of their Web platform started by improving their online search engine, but as William Greenly mentioned, it quickly became a strategic decision:

“So the objectives were about site search and improving it, but in the long-run it was always the idea to contextualize content, to facet content, to promote it in different contexts.”

The objective is to create a platform that gives them the flexibility to leverage all the data assets they own. This flexibility will help them to leverage the data assests they have to improve not only their search engine, but also to contextualize it in different parts of their websites, partner’s websites or to promote, and publish that same information on different communication channels or devices.

The Flexibility

What is a flexible platform in that context? A flexible platform is one that can integrate any kind of information sources. Such information sources in the context of Volkswagen can be a series of relational dataset schemas spread around the World, Excel spreadsheets, CSV files, old plain text technical documents about past model of cars, semi-structured documents such as webpages, etc.

A flexible platform is also one that minimally impact (if at all) the data consumers if the data structure changes in the system. This is really important since the World we live in constantly changes. This means that things constantly change and we have to reflect these changes in the data we own and maintain. This is why this point is so important, because we want to minimize the impact of the data structure changes that will happen all the time.

Having the flexibility to constantly adapt your data, while minimally impacting the data consumers of the system, enables you to make quick decision to adapt your strategy in a highly competitive World. This flexibility gives you a clear business advantage.

A flexible platform is also one that let you publish your data the way you want, in the format that is needed. Such a flexible platform has to give you access to an interface that give you access to all the functionalities of the platform without having to care about what happens under the hood.

A flexible system is one that can communicate your information on any kind of communication channels, and to any devices that have access to the Web.

Under the Hood

That next generation platform that Volkswagen is currently developing is partly based on a few of the main pieces of the Open Semantic Framework. These pieces help them to reach their goal by helping them giving the flexibility their platform needs.

The first step they gone thru was to create their Volkswagen Vehicles Ontology that is used to describe all the entities they want to index into their platform. The Web Ontology Language (OWL), along with the Resource Description Framework (RDF) is what gives them the complete flexibility on how they can integrate all the pieces of information they want, in a canonical format.

Then they choose to use structWSF (the structured data web services framework). This piece gives them the flexibility to get a series of web interfaces (web service endpoints) to create, update, manage and query their data. This web service layer enables them to do anything they want with their data, from anywhere on the Web. This is possible because all the functionalities of the framework are exposed as web service endpoints. StructWSF also gives them the possibility to communicate their data in multiple different formats. This makes it the perfect flexible system to feed their information in different contexts, in different communication channels or on different devices.

At Volkswagen, structWSF is used to populate, and keep in sync, their Solr and Triple Store instances. It gives them the time to care about the more important aspects of their platform, and to care about how the data should be synced between the various specialized data management systems.

By using structWSF to manage their data, they are able to reach some objectives to make their platform as flexible as possible:

  • To be able to minimize the impact of data changes to the data consumers
    • Because structWSF uses OWL & RDF to describe all the data it index
  • To be able to manipulate their data from anywhere
    • Because all the functionalities of structWSF are exposed as web service endpoints
  • To be able to communicate the information in different contexts, communication channels and devices
    • Because structWSF has, in its core, is designed to transform all the data it indexes in any other kind of format

The Next Step

One of their longer term goal and objective is to analyze their unstructured and semi-structured textual documents to extract some structure out of them, and to index them into their semantic platform. To do this, they are looking at using Scones, which is the structWSF semantic tagger web service endpoint. Scones will use some subject reference structures such as UMBEL to semantically tag the textual document. Once the document as been processed by Scones, and indexed in structWSF, it can now be re-published in different contexts based on the reference concepts that have been tagged to it. This gives them the flexibility to leverage non-structured sources of data and to re-purpose it in different ways by publishing it in different context and in different systems.

This second system will enable them to leverage the investment they made in the past, by writing all these textual documents, and to re-purpose, and re-contextualizing, them in all kind of different contexts.

Conclusion

I think that TribalDDB and Volkswagen make the good decision for their future. Taking the business decision to develop and maintain a completely new kind of information system is not an easy decision to take. I am not saying that they made the good choice to use our pieces of the stack. The decision goes far beyond this. Such a Semantic Platform challenges everything in an organization: the people that takes the decisions, the people that create and manage the data, the people that develop the system, the people that maintain that system, the consumers of the system, the customers, the partners, etc. This is a big decision; whatever the technology stack you plan to use. I congratulate them for the decision they took.

I strongly believe that this was the right decision to take considering the future opportunities they are creating to themselves.

 

 

Querying the MyPeg datasets using the structWSF SPARQL endpoint

Frederick Giasson
The last blog post I wrote demonstrated how one could query the MyPeg.ca portal using the full set of structWSF web service endpoints to get data out of the portal. However, I didn’t cover the usage of the SPARQL endpoint since I wanted to cover it in its own blog post to explain all its characteristics.

In this blog post, I will demonstrate how one can get data out of the MyPeg.ca community indicators web portal for Winnipeg’s citizens using the SPARQL endpoint. I will also cover all the specificities of this SPARQL endpoint: all its characteristics and access/permission features.

Two Modes

The first characteristic of the structWSF SPARQL endpoint is that there are two modes (usecases) it can be used for:

  1. Getting SPARQL resultsets that match some SPARQL queries patterns
  2. Getting complete records descriptions in any format supported by the endpoint

The first mode is not different than any other SPARQL endpoint. Users just send different SPARQL queries and retrieve their related SPARQL resultsets. These resultsets can be returned using different MIME types.

For a SELECT query, these formats can be used:

  1. application/sparql-results+xml
  2. application/sparql-results+json

For a DESCRIBE or a CONSTRUCT query, these formats can be used:

  1. text/rdf+n3 (RDF+N3)
  2. application/rdf+xml (RDF+XML)
  3. application/rdf+json (RDF+JSON)
  4. text/plain (NTRIPLES)

The second mode is quite different. The main characteristic of the structWSF SPARQL endpoint is that it can export resultsets into different formats, not usually supported by other endpoints. However, these other formats are usually used to describe complete descriptions of records, and not just some triples matching some SPARQL patterns.

For that reason, the SPARQL query that is sent using this second mode needs to have the three variables (1) ?s, (2) ?p and (3) ?o bound in the SPARQL query, otherwise an empty resultset will be returned. For example, the following SPARQL query would return the complete records descriptions of all the records that are peg:Theme and that are themes of the peg:WellBeing cross cutting issue:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]SELECT ?s ?p ?o
WHERE
{
?s a <http://purl.org/ontology/peg#Theme> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> ;
?p ?o .
}[/cc]

This mode is used to return a set of records descriptions that match a SPARQL pattern. Supported MIME formats for that second mode are:

  • text/xml (structXML)
  • application/json (structXML in JSON)
  • application/rdf+xml (RDF+XML)
  • application/rdf+n3 (RDF+N3)
  • application/sparql-results+xml (SPARQL resultset in XML)
  • application/sparql-results+json (SPARQL resultset in XML)

Getting Records in Different Formats

Now, let’s take a look at what is returned for the SPARQL query above, for each of these supported MIME types, from the MyPeg.ca SPARQL endpoint.

Note that the queries below are using the Curl application (available for multiple operating systems) to send the HTTP queries to the structWSF SPARQL web service endpoint.

StructXML: text/xml

[cc lang=’text’ line_numbers=’false’ escaped=’true’]
curl -H “Accept: text/xml” “http://www.mypeg.ca/ws/sparql/” -d “dataset=http://www.mypeg%2Eca%2Fwsf%2Fdatasets%2F249%2F&query= SELECT+%3Fs+%3Fp+%3Fo%0D%0AWHERE%0D%0A{%0D%0A++%3Fs+a+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23Theme>+%3B%0D%0A+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23isThemeOf>+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing>+%3B%0D%0A+++++%3Fp+%3Fo+.%0D%0A}%0D%0A”
[/cc]

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]

<?xml version=”1.0″ encoding=”utf-8″?>
<resultset>
<prefix entity=”owl” uri=”http://www.w3.org/2002/07/owl#”/>
<prefix entity=”rdf” uri=”http://www.w3.org/1999/02/22-rdf-syntax-ns#”/>
<prefix entity=”rdfs” uri=”http://www.w3.org/2000/01/rdf-schema#”/>
<prefix entity=”wsf” uri=”http://purl.org/ontology/wsf#”/>
<subject type=”http://purl.org/ontology/peg#Theme” uri=”http://purl.org/ontology/peg/framework#Economy”>
<predicate type=”http://purl.org/ontology/peg#isThemeOf”>
<object uri=”http://purl.org/ontology/peg/framework#WellBeing”/>
</predicate>
<predicate type=”http://purl.org/ontology/peg#isThemeOf”>
<object uri=”http://purl.org/ontology/peg/framework#Poverty”/>
</predicate>
<predicate type=”http://purl.org/ontology/sco#displayComponent”>
<object uri=”http://purl.org/ontology/sco#sRelationBrowser”/>
</predicate>
<predicate type=”http://www.w3.org/2000/01/rdf-schema#label”>
<object type=”rdfs:Literal”>economy</object>
</predicate>
<predicate type=”http://purl.org/ontology/iron#prefLabel”>
<object type=”rdfs:Literal”>Economy</object>
</predicate>
<predicate type=”http://purl.org/dc/elements/1.1/description”>
<object type=”rdfs:Literal”>Economy includes all that people do in our community to produce, exchange, distribute, and consume goods and services.</object>
</predicate>
</subject>
</resultset>

[/cc]

StructXML in JSON: application/json

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

curl -H “Accept: application/json” “http://www.mypeg.ca/ws/sparql/” -d “dataset=http://www.mypeg%2Eca%2Fwsf%2Fdatasets%2F249%2F&query= SELECT+%3Fs+%3Fp+%3Fo%0D%0AWHERE%0D%0A{%0D%0A++%3Fs+a+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23Theme>+%3B%0D%0A+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23isThemeOf>+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing>+%3B%0D%0A+++++%3Fp+%3Fo+.%0D%0A}%0D%0A”

[/cc]

[cc lang=’javascript’ line_numbers=’false’ escaped=’true’]

{
“prefixes”: [
{
“owl”: “http://www.w3.org/2002/07/owl#”,
“rdf”: “http://www.w3.org/1999/02/22-rdf-syntax-ns#”,
“rdfs”: “http://www.w3.org/2000/01/rdf-schema#”,
“wsf”: “http://purl.org/ontology/wsf#”,
“ns0”: “http://purl.org/ontology/peg#”,
“ns1”: “http://purl.org/ontology/sco#”,
“ns2”: “http://purl.org/ontology/iron#”,
“ns3”: “http://purl.org/dc/elements/1.1/”
}
],
“resultset”: {
“subject”: [
{
“uri”: “http://purl.org/ontology/peg/framework#Economy”,
“type”: “ns0:Theme”,
“predicates”: [
{
“ns0:isThemeOf”: {
“uri”: “http://purl.org/ontology/peg/framework#WellBeing”
}
},
{
“ns0:isThemeOf”: {
“uri”: “http://purl.org/ontology/peg/framework#Poverty”
}
},
{
“ns1:displayComponent”: {
“uri”: “http://purl.org/ontology/sco#sRelationBrowser”
}
},
{
“rdfs:label”: “economy”
},
{
“ns2:prefLabel”: “Economy”
},
{
“ns3:description”: “Economy includes all that people do in our community to produce, exchange, distribute, and consume goods and services.”
}
]
},
]
}
}

[/cc]

RDF in XML: application/rdf+xml

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

curl -H “Accept: application/rdf+xml” “http://www.mypeg.ca/ws/sparql/” -d “dataset=http://www.mypeg%2Eca%2Fwsf%2Fdatasets%2F249%2F&query= SELECT+%3Fs+%3Fp+%3Fo%0D%0AWHERE%0D%0A{%0D%0A++%3Fs+a+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23Theme>+%3B%0D%0A+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23isThemeOf>+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing>+%3B%0D%0A+++++%3Fp+%3Fo+.%0D%0A}%0D%0A”

[/cc]

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]

<?xml version=”1.0″?>
<rdf:RDF  xmlns:owl=”http://www.w3.org/2002/07/owl#” xmlns:rdf=”http://www.w3.org/1999/02/22-rdf-syntax-ns#” xmlns:rdfs=”http://www.w3.org/2000/01/rdf-schema#” xmlns:wsf=”http://purl.org/ontology/wsf#” xmlns:ns0=”http://purl.org/ontology/peg#” xmlns:ns1=”http://purl.org/ontology/sco#” xmlns:ns2=”http://purl.org/ontology/iron#” xmlns:ns3=”http://purl.org/dc/elements/1.1/”>

<ns0:Theme rdf:about=”http://purl.org/ontology/peg/framework#Economy”>
<ns0:isThemeOf rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing” />
<ns0:isThemeOf rdf:resource=”http://purl.org/ontology/peg/framework#Poverty” />
<ns1:displayComponent rdf:resource=”http://purl.org/ontology/sco#sRelationBrowser” />
<rdfs:label>economy</rdfs:label>
<ns2:prefLabel>Economy</ns2:prefLabel>
<ns3:description>Economy includes all that people do in our community to produce, exchange, distribute, and consume goods and services.</ns3:description>
</ns0:Theme>

</rdf:RDF>

[/cc]

RDF in N3: application/rdf+n3

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

curl -H “Accept: application/rdf+n3” “http://www.mypeg.ca/ws/sparql/” -d “dataset=http://www.mypeg%2Eca%2Fwsf%2Fdatasets%2F249%2F&query= SELECT+%3Fs+%3Fp+%3Fo%0D%0AWHERE%0D%0A{%0D%0A++%3Fs+a+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23Theme>+%3B%0D%0A+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23isThemeOf>+<http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing>+%3B%0D%0A+++++%3Fp+%3Fo+.%0D%0A}%0D%0A”

[/cc]

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix wsf: <http://purl.org/ontology/wsf#> .

<http://purl.org/ontology/peg/framework#Economy> a <http://purl.org/ontology/peg#Theme> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#Poverty> ;
<http://purl.org/ontology/sco#displayComponent> <http://purl.org/ontology/sco#sRelationBrowser> ;
<http://www.w3.org/2000/01/rdf-schema#label> “””economy””” ;
<http://purl.org/ontology/iron#prefLabel> “””Economy””” ;
<http://purl.org/dc/elements/1.1/description> “””Economy includes all that people do in our community to produce, exchange, distribute, and consume goods and services.””” .

[/cc]

Getting Records Using CONSTRUCT

You always have the possibility to use a CONSTRUCT query to return data in different formats. Unlike with the second mode supported by the endpoint, you won’t have access to different formats (such as structXML both in XML and JSON). Here is such a CONSTRUCT query:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]
CONSTRUCT FROM <http://www.mypeg.ca/wsf/datasets/249/>
{
?s ?p ?o .
}
WHERE
{
?s peg:isThemeOf <http://purl.org/ontology/peg/framework#WellBeing> .
?s ?o ?p .
}
[/cc]

[cc lang=’text’ line_numbers=’false’ escaped=’true’]
curl -H “Accept: application/rdf+xml” “http://www.mypeg.ca/ws/sparql/” -d “query=PREFIX%20peg%3A%20%3Chttp%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23%3E%0ACONSTRUCT%0A%7B%0A%20%20%3Fs%20peg%3AisThemeOf%20%3Chttp%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing%3E%20.%0A%7D%0AFROM%20%3Chttp%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F249%2F%3E%0AWHERE%20%0A%7B%20%0A%20%20%3Fs%20peg%3AisThemeOf%20%3Chttp%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing%3E%20.%0A%7D”
[/cc]

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]
<?xml version=”1.0″ encoding=”utf-8″ ?>
<rdf:RDF xmlns:rdf=”http://www.w3.org/1999/02/22-rdf-syntax-ns#” xmlns:rdfs=”http://www.w3.org/2000/01/rdf-schema#”>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#EducationAndLearning”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#BasicNeeds”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#Health”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#SocialVitality”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#BuiltEnvironment”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#NaturalEnvironment”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#Economy”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
<rdf:Description rdf:about=”http://purl.org/ontology/peg/framework#Governance”><n0pred:isThemeOf xmlns:n0pred=”http://purl.org/ontology/peg#” rdf:resource=”http://purl.org/ontology/peg/framework#WellBeing”/></rdf:Description>
</rdf:RDF>
[/cc]

SPARQL Queries Restrictions

The structWSF SPARQL endpoint has some restrictions that have been introduced to make sure that the requesting users can only query the data to which they have access.

In structWSF, all permissions are attached to a dataset (a graph). Different users have different Create, Read, Update and Delete permissions on different datasets hosted on the same structWSF endpoint. Because of this core mechanism in structWSF, we had to make sure that these same restrictions were applied for the SPARQL endpoint. This means that different SPARQL clauses and usages are restricted.

This section covers these specific restrictions for a structWSF SPARQL endpoint.

Accessing Dataset Without Permissions

Let’s try to see what happens when someone tries to access a dataset to which he doesn’t have access. Consider this SPARQL query:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]
PREFIX mypeg: <http://www.mypeg.ca/wsf/>
SELECT ?s ?p ?o FROM mypeg:
WHERE
{
?s ?p ?o .
}
[/cc]

Obviously, no user has a direct access to that dataset on the MyPeg instance:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]
curl -H “Accept: application/rdf+n3” “http://www.mypeg.ca/ws/sparql/” -d “query= PREFIX%20mypeg%3A%20%3Chttp%3A%2F%2Fwww.mypeg.ca%2Fwsf%2F%3E%0ASELECT%20%3Fs%20%3Fp%20%3Fo%20FROM%20mypeg%3A%0AWHERE%0A%7B%0A%20%20%3Fs%20%3Fp%20%3Fo%20.%0A%7D%0A”
[/cc]

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]

<error>
<id>WS-AUTH-VALIDATOR-303</id>
<webservice>/ws/auth/validator/</webservice>
<name>No access defined</name>
<description>No access defined for this requester IP , dataset and web service</description>
<debugInformation>No access defined for this requester IP (174.129.43.163), dataset (http://www.mypeg.ca/wsf/) and web service (http://www.mypeg.ca/wsf/ws/sparql/)</debugInformation>
<level>Warning</level>
</error>

[/cc]

So, even if a dataset exists in a triple store that exposes a SPARQL endpoint, not all users have access to all of these datasets. The access and permissions layer will restrict the access to them if need be.

If a FROM clause, or multiple FROM NAMED clauses are specified in the SPARQL query, the access layer will make sure that the user has access to all these datasets. If he doesn’t have access to one of them, then an error will be returned.

CONSTRUCT

The CONSTRUCT clause can be used against this SPARQL endpoint, but only if it doesn’t use any GRAPH clauses. However, we encourage users to use the method described in the section “Getting Records in Different Formats” since more formats can be requested, and more formats can easily be added in the future.

Here is an example of a CONSTRUCT query that uses a GRAPH clause:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]
CONSTRUCT
{
?s ?p ?o
}
WHERE
{
graph <http://www.mypeg.ca/wsf/datasets/249/>
{
?s ?p ?o
}
}
[/cc]

[cc lang=’text’ line_numbers=’false’ escaped=’true’]
curl -H “Accept: application/rdf+xml” “http://www.mypeg.ca/ws/sparql/” -d ” query=CONSTRUCT%0A%7B%0A%20%20%3Fs%20%3Fp%20%3Fo%0A%7D%0AWHERE%20%0A%7B%20%0A%20%20graph%20%3C%20http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F249%2F%3E%0A%20%20%7B%0A%20%20%20%20%3Fs%20%3Fp%20%3Fo%0A%20%20%7D%0A%7D%0A”
[/cc]

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]
<?xml version=”1.0″ encoding=”utf-8″?>
<error>
<id>WS-SPARQL-205</id>
<webservice>/ws/sparql/</webservice>
<name>GRAPH not permitted.</name>
<description>The SPARQL GRAPH clause is not permitted for this sparql endpoint. Please change your SPARQL query to specify the datasets you want to query with the FROM and FROM NAMED sparql clauses, or with the dataset parameter.</description>
<debugInformation></debugInformation>
<level>Warning</level>
</error>
[/cc]

As you can see, the endpoint will return a 205 error if a GRAPH clause is used within a CONSTRUCT statement.

GRAPH

As we saw above, no GRAPH clauses can be used in a SPARQL query. The reason is that we don’t want people to send SPARQL queries with GRAPH clauses that use variables. Otherwise, if we permitted GRAPH clauses to be used with variables, we couldn’t currently determine what triple comes from what dataset and so, we couldn’t ensure the access and permissions to that data.

However, in the future two improvements could be created to enable the usage of GRAPH clauses in SPARQL queries processed by structWSF:

  1. We could enable people to use GRAPH clauses that use direct IRI_REF references. That way, structWSF could easily check the permissions for these graphs (just like it does handle the FROM, FROM NAMED and DESCRIPTION clauses).
  2. We could enable the full usage of the GRAPH clause. However, we would have to modify the queries at the level of the endpoint to get the graph provenance of all the triples. Then the endpoint would have to analyze the provenance of each triple and only return the ones that the user has access to. This would inevitably slow down the query time to process the SPARQL request.

In the mean time, no GRAPH clauses can be used in any SPARQL query, and people should use the FROM and FROM NAMED clauses to get access to all the datasets they want from a particular endpoint.

SPARUL

No SPARQL/Update (SPARUL) queries can be sent via the structWSF SPARQL endpoint. All data modifications (records and/or dataset creation, updating and deleting) have to be performed by the Dataset and Record CRUD web service endpoints.

Conclusion

The structWSF SPARQL endpoint is a wrapper above a triple store’s SPARQL endpoint. It adds a permissions and access layer that is compatible with that used by other structWSF web services. This permission layer ensures that requesters only access the information they have access to within the triple store. Also, all of these access permissions are managed by the other structWSF web service endpoints, and can also be managed via the conStruct user interface.

The structWSF SPARQL endpoint also supports more resultset formats than are generally supported by mainstream triple stores. Also, the addition of new formats is made easier by using structWSF’s way to convert data in different formats.

Getting Data Out of MyPeg.ca using structWSF Endpoints

Frederick Giasson
A few weeks ago I presented the new MyPeg.ca community indicators web portal for Winnipeg’s citizens. I explained how in MyPeg.ca we leverage Structured Dynamics’ semantic technologies stack (akaThe Semantic Muffin). Today’s blog post explains one facet of the project that shows how external agents (people, services, software, etc.) can interact with the system’s indicator datasets using the structWSF web service endpoints.Since this post focuses only on data export, I suggest you read the structWSF Web Services Tutorial for a complete overview of how the endpoints architecture works.

Merging Pipes

Two Main structWSF Characteristics: Accessibility & Management

structWSF is a set of 22 web service endpoints that lets you integrate data from different sources, manage that integrated data, and publish it via different communication channels such as web pages, software applications, etc.

Obviously, the main characteristic of this framework is that everything is a web service. This means that all functionality of the system can be accessed from anywhere on the Internet. However, this doesn’t mean that everything is open like a snack-bar. In fact, there are two levels of accessibility: (1) access to the web service endpoint’s URL, and (2) access to the content of datasets hosted on structWSF. Depending on the usecase, people could restrict the direct access to the web service endpoint(s) by properly configuring their web server, others could choose to let anyone access the endpoints, but would restrict the access to the dataset(s) hosted by structWSF. In case of MyPeg.ca, the sponsor chose to open the access to their web service endpoints and datasets.

Just by surfing on the MyPeg.ca portal, you are already leveraging these endpoints in multiple different ways. First, each time you generate a browse or a search Web page, you are telling the web server to send multiple queries to different endpoints; then the web page’s content will be populated with that information and presented to you. But, each time you click on an explorer node, your web browser is also sending queries to exactly the same web service endpoints. So, in one case a PHP script acts to query the endpoints; and, in other cases, a Flash Semantic Component does. Depending, all structWSF data can be accessed from quite different environments.

The other main characteristic of structWSF is that any kind of data can be imported in, and exported out, of the system. structWSF leverages RDF (Resource Description Framework) as the canonical data format that can be used to express any other formats. It is because of the usage of RDF that structWSF can act as an effective ETL (Extract, transform, load) system. Depending on the web service endpoint, the output formats currently supported by most of the endpoints are:

But the architecture of the web service endpoints can easily accommodate other formats if needed for a specific usecase.

Getting Data Out Of MyPeg.ca

Now, how can you get data out of MyPeg.ca? There are really two methods. This blog post discusses the CRUD: Read, Browse and Search web service endpoints. In my next blog post, I will focus on using the SPARQL web service endpoint to do the same.

All of the query examples in this blog post will use a tool called Curl to send the queries and to get back the resultsets. I encourage you to download and use that tool to test these endpoints and to gain a feeling for how it works. Also note that only the first record of each resultset is recorded below (of course, the actual results include all records).

Browse

The Browse web service endpoint is used to return lists of records. These records can also be filtered according to their provenance (dataset), type and the attributes that describe them. Now, let’s see how you can use this web service to get data out of MyPeg.ca.

First, there are three datasets available to the public:

  1. Well-being Indicators (http://www.mypeg.ca/wsf/datasets/258/)
  2. Stories (http://www.mypeg.ca/wsf/datasets/272/)
  3. PEG Framework (http://www.mypeg.ca/wsf/datasets/249/)

The resultsets can be serialized using one of these four different formats:

  • text/xml (structXML)
  • application/json (structXML in JSON)
  • application/rdf+xml (RDF/XML)
  • application/rdf+n3 (RDF/N3)

Note: if one of your desired formats is not directly available at the endpoint level, you can always use one of the converter web service endpoints such as: commON, irJSON or TSV/CSV.

Get the first 10 results of the Stories dataset in structXML

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: text/xml” “http://www.mypeg.ca/ws/browse/” -d “attributes=all&types=all&datasets=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2F&items=10&page=0&inference=on&include_aggregates=true”[/cc]

StructXML resultset:

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]

<?xml version=”1.0″ encoding=”utf-8″?>
<resultset>
<prefix entity=”owl” uri=”http://www.w3.org/2002/07/owl#”/>
<prefix entity=”rdf” uri=”http://www.w3.org/1999/02/22-rdf-syntax-ns#”/>
<prefix entity=”rdfs” uri=”http://www.w3.org/2000/01/rdf-schema#”/>
<prefix entity=”wsf” uri=”http://purl.org/ontology/wsf#”/>
<subject type=”http://purl.org/ontology/muni#Story” uri=”http://www.mypeg.ca/wsf/datasets/272/resource/AgeOpportunity”>
<predicate type=”http://purl.org/dc/terms/isPartOf”>
<object type=”http://rdfs.org/ns/void#Dataset” uri=”http://www.mypeg.ca/wsf/datasets/272/”/>
</predicate>
<predicate type=”http://purl.org/ontology/iron#prefLabel”>
<object type=”rdfs:Literal”>Age &amp; Opportunity</object>
</predicate>
<predicate type=”http://purl.org/dc/terms/created”>
<object type=”rdfs:Literal”>2010-10-28T19:38:58+00:00</object>
</predicate>
<predicate type=”http://purl.org/ontology/bibo/abstract”>
<object type=”rdfs:Literal”>Amanda Macrae, Deborah Lorteau and Stacey Miller work for Age and Opportunity.
The majority of clients are older adults living at lower socio economic status. When addressing the housing issue they say, “In a nutshell, it’s dire.” There is simply not enou…</object>
</predicate>
<predicate type=”http://purl.org/ontology/peg#interviewee”>
<object type=”rdfs:Literal”>Amanda Macrae, Deborah Lorteau, Stacey Miller</object>
</predicate>
<predicate type=”http://purl.org/ontology/peg#interviewer”>
<object type=”rdfs:Literal”>Molly Johnson</object>
</predicate>
<predicate type=”http://purl.org/ontology/peg#storyRelatedAgencyProgram”>
<object type=”rdfs:Literal”>Age &amp; Opportunity</object>
</predicate>
<predicate type=”http://purl.org/ontology/sco#storyAnnotatedTextUri”>
<object>http://www.mypeg.ca/scones/AgeOpportunity.xml</object>
</predicate>
<predicate type=”http://purl.org/ontology/sco#storyTextUri”>
<object type=”rdfs:Literal”>http://www.mypeg.ca/scones/AgeOpportunity.txt</object>
</predicate>
</subject>
</resultset>

[/cc]

Get the 10 first results from all datasets that are records of type Neighborhoods in RDF/XML

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/rdf+xml ” “http://www.mypeg.ca/ws/browse/” -d “attributes=all& type=http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%23Neighborhood &datasets=all&items=10&page=0&inference=on&include_aggregates=true”[/cc]

RDF/XML resultset:

[cc lang=’xml’ line_numbers=’false’ escaped=’true’]

<?xml version=”1.0″?>
<rdf:RDF  xmlns:owl=”http://www.w3.org/2002/07/owl#” xmlns:rdf=”http://www.w3.org/1999/02/22-rdf-syntax-ns#” xmlns:rdfs=”http://www.w3.org/2000/01/rdf-schema#” xmlns:wsf=”http://purl.org/ontology/wsf#” xmlns:ns0=”http://purl.org/ontology/peg#” xmlns:ns1=”http://purl.org/dc/terms/” xmlns:ns2=”http://purl.org/ontology/iron#” xmlns:ns3=”” xmlns:ns4=”http://purl.org/dc/elements/1.1/” xmlns:ns5=”http://purl.org/ontology/aggregate#”>

<ns0:Component rdf:about=”http://purl.org/ontology/peg/framework#Safety”>
<ns1:isPartOf rdf:resource=”http://www.mypeg.ca/wsf/datasets/249/” />
<ns2:prefLabel>Safety</ns2:prefLabel>
<ns2:altLabel>safety</ns2:altLabel>
<ns3:>safety</ns3:>
<ns4:description>Safety is the state of being “safe”, the condition of being protected against physical, social, spiritual, financial, political, emotional, occupational, psychological, educational or other types or consequences of failure, damage, error, accidents, harm or any other event which could be considered non-desirable.</ns4:description>
<rdfs:comment>Includes the idea of safety prevention</rdfs:comment>
<rdfs:seeAlso>http://en.wikipedia.org/wiki/Safety</rdfs:seeAlso>
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#HouseholdIncome” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#LowIncomeCutOffAfterTax” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#MarketBasketMeasure” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#ParticipationInSportsAndRecreation” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#MaternalSocialIsolation” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#PersonalSafety” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#EarlyDevelopmentInstrument” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#HighSchoolGraduationRate” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#LongTermUnemployment” />
<ns0:hasIndicator rdf:resource=”http://purl.org/ontology/peg/framework#TeenageBirths” />
<ns0:isComponentOf rdf:resource=”http://purl.org/ontology/peg/framework#BasicNeeds” />
<ns0:isComponentOf rdf:resource=”http://purl.org/ontology/peg/framework#Poverty” />
</ns0:Component>
</rdf:RDF>

[/cc]

Search

The Search web service endpoint is also used to return lists of records. These records should match a search string and can also be filtered according to their provenance (dataset), type and the attributes that describe them.

The same mime types and datasets as the ones for the Browse web service are available for the Search endpoint.

Searching for records with the keyword “poverty” and get resultsets in RDF/N3

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/rdf+n3” “http://www.mypeg.ca/ws/search/” -d “query=poverty&datasets=all&items=10&page=0&inference=on&include_aggregates=true”[/cc]

RDF/N3 resultset:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix wsf: <http://purl.org/ontology/wsf#> .

<http://purl.org/ontology/peg/framework#Poverty> a <http://purl.org/ontology/peg#CrossCuttingIssue> ;
<http://purl.org/dc/terms/isPartOf> <http://www.mypeg.ca/wsf/datasets/249/> ;
<http://purl.org/ontology/iron#prefLabel> “””Poverty””” ;
<http://purl.org/dc/elements/1.1/description> “””Poverty is not having the sufficient resources, capabilities, choices, security and power necessary to enjoy an adequate standard of living.  Poverty includes much more than a lack of money.  It includes being excluded from ordinary living patterns, customs and activities.  Consequently, people living in poverty are often unable to participate fully in their communities or to reach their full potential.””” ;
<http://www.w3.org/2000/01/rdf-schema#seeAlso> “””http://en.wikipedia.org/wiki/Poverty””” .

[/cc]

CRUD: Read

The Browse and Search web service endpoints are really used to find lists of records according to some provided criteria. However, the complete description of these records is not returned by these endpoints, but only the information necessary to create the proper list to display to users in a user interface. So, to get the complete description of a record (or multiples thereof), you have to use the CRUD: Read web service endpoint. Also, sometimes you may get a reference to a record hosted on a structWSF node, then CRUD: Read is the way to get its full description.

Get the full description of the Ida story in irJSON

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/iron+json” “http://www.mypeg.ca/ws/crud/read/?uri=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2Fresource%2FIda&dataset=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F272%2F&include_reification=true&include_linksback=false[/cc]

irJSON resulset:

[cc lang=’javascript’ line_numbers=’false’ escaped=’true’]

{
“dataset”: {
“linkage”: [
{
“linkedType”: “application/rdf+xml”,
“attributeList”: {
“created”: {
“mapTo”: “http://purl.org/dc/terms/created”
},
“isAbout”: {
“mapTo”: “http://umbel.org/umbel#isAbout”
},
“prefLabel”: {
“mapTo”: “http://purl.org/ontology/iron#prefLabel”
},
“interviewee”: {
“mapTo”: “http://purl.org/ontology/peg#interviewee”
},
“interviewer”: {
“mapTo”: “http://purl.org/ontology/peg#interviewer”
},
“abstract”: {
“mapTo”: “http://purl.org/ontology/bibo/abstract”
},
“storyVideoAudio”: {
“mapTo”: “http://purl.org/ontology/peg#storyVideoAudio”
},
“storyAnnotatedTextUri”: {
“mapTo”: “http://purl.org/ontology/sco#storyAnnotatedTextUri”
},
“storyTextUri”: {
“mapTo”: “http://purl.org/ontology/sco#storyTextUri”
}
},
“typeList”: {
“Story”: {
“mapTo”: “http://purl.org/ontology/muni#Story”
}
}
}
]},
“recordList”: [
{
“id”: “http://www.mypeg.ca/wsf/datasets/272/resource/Ida”,
“type”: “Story”,
“created”: “2010-10-28T18:11:27+00:00”,
“isAbout”: [
{
“ref”: “@@http://purl.org/ontology/peg/framework#EducationAndLearning”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Health”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Program”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Income”
},
{
“ref”: “@@http://purl.org/ontology/peg/framework#Poverty”
}     ],
“prefLabel”: “Ida”,
“interviewee”: “Ida”,
“interviewer”: “Christa Rust”,
“abstract”: “‘Poverty is earning just enough to get by; never having money for extras.’\n\nIda is the mother of two grown children, eight years apart.  She lives in a small bachelor suite, which costs her $428 per month, or 62% of her income.  She volunteers twice a we…”,
“storyVideoAudio”: “http://www.youtube.com/v/0zIqtYhiHfM”,
“storyAnnotatedTextUri”: “http://www.mypeg.ca/scones/Ida.xml”,
“storyTextUri”: “http://www.mypeg.ca/scones/Ida.txt”
}
]
}

[/cc]

Get Well-Being record description with linkbacks in RDF+N3

The characteristic of this query is that I enabled the “include_linksback” parameter. This returns a reference to all the records, in the datasets hosted on the structWSF node, that refers to that target record.

Query:

[cc lang=’text’ line_numbers=’false’]curl -H “Accept: application/rdf+n3” “http://www.mypeg.ca/ws/crud/read/?uri=http%3A%2F%2Fpurl.org%2Fontology%2Fpeg%2Fframework%23WellBeing&datast=http%3A%2F%2Fwww.mypeg.ca%2Fwsf%2Fdatasets%2F249%2F&registered_ip=self%3A%3A0&include_reification=true&include_linksback=true”[/cc]

RDF+N3 resultset:

[cc lang=’text’ line_numbers=’false’ escaped=’true’]

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

<http://purl.org/ontology/peg/framework#WellBeing> a <http://purl.org/ontology/peg/framework#WellBeing> ;
<http://purl.org/ontology/iron#prefLabel> “””Well-being””” ;
<http://purl.org/dc/elements/1.1/description> “””Well-being refers to the general quality of life experienced by individuals and communities. The elements of wellbeing include: the ability to meet basic needs, the economy, health, the built environment, governance, education and learning, the natural environment, and social vitality.””” ;
<http://purl.org/ontology/sco#displayComponent> <http://purl.org/ontology/sco#sRelationBrowser> .

<http://purl.org/ontology/peg/framework#WellBeing> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Economy> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Governance> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#BuiltEnvironment> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#NaturalEnvironment> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#SocialVitality> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#BasicNeeds> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#EducationAndLearning> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

<http://purl.org/ontology/peg/framework#Health> a <http://www.w3.org/2002/07/owl#Thing> ;
<http://purl.org/ontology/peg#isThemeOf> <http://purl.org/ontology/peg/framework#WellBeing> .

[/cc]

General Endpoint Parameters

The general parameters available for each of these web services is provided in their respective TechWiki documentation. For that detailed information, see the Browse, Search, or CRUD: Read articles.

Conclusion

As you can see, agents can get different kinds of data from the MyPeg.ca portal by querying a set of web service endpoints. This is one way to get data out of the system. These data can then be accessed for indexing in other systems, for direct use, or for dynamic applications like browsing the nodes in the explorer.

This is one of the ways to get data out of the system. A user can also export that same information from the Export features on the Browse, Search and Record View pages. Also, other methods will be explained in the next blog posts from this MyPeg.ca series.

All in all, this shows how effective structWSF can be to integrate, manage and publish a wide range of data in different data formats. It also shows how completely different parts of your software architecture can leverage your information, the way you want, from anywhere on the Internet.

Building conStruct Templates

Frederick Giasson
The templating engine in conStructis one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

  • Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.
  • Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.
  • Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on “text”, you will get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.


As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

 

Building conStruct Templates

The templating engine in conStruct is one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…

Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

�Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.

�Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.

�Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on �text�, you will

Building conStruct Templates

The templating engine in conStruct is one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…

Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

  • Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.
  • Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.
  • Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on �text�, you will get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.

As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.

As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!