structFieldStorage: A New Field Storage System for Drupal 7

Frederick Giasson

Structured Dynamics has been working with Drupal for quite some time. This week marks our third anniversary of posting code to the contributed conStruct modules in Drupal. But, what I’m able to share today is our most exciting interaction with Drupal to date. In essence, we now can run Drupal directly from an RDF triplestore and take full advantage of our broader Open Semantic Framework (OSF) stack. Massively cool!

On a vanilla Drupal 7 instance, everything ends up being saved into Drupal’s default storage system. This blog post introduces a new way to save (local) Content Type entities: the structfieldstorage field storage system. This new field storage system gives the possibility to Drupal administrators to choose to save specific (or all) fields and their values into a remote structWSF instance. This option replaces Drupal’s default storage system (often MySQL) for the content types and their fields chosen.

By using this new field storage system, all of the local Drupal 7 content can be queried via any of structWSF’s web service endpoints (which includes a SPARQL endpoint). This means that all Drupal 7 content (using this new storage system) gets converted and indexed as RDF data. This means that all of the Drupal local content gets indexed in a semantic web service framework.

Fields and Bundles

There are multiple core concepts in Drupal, two of which are Bundles and Fields. A Field is basically an attribute/value tuple that describes an entity. A Bundle is a set (an aggregation) of fields. The main topic of this blog post is a special feature of the field: their storage system.

In Drupal, each field instance does have its own field storage system associated to it. A field storage system is a system that manages the field/value tuples of each entity that has been defined as a Drupal instance. The default storage system of any field is the field_sql_storage, which is normally a MySQL server or database.

The field storage system allows a bundle to have multiple field instances, each of which may have a different field storage target. This means that the data that describes an entity can be saved in multiple different data stores. Though it may appear odd at first as to why such flexibility has merit, but we will see that this design is quite clever, and probably essential.

There are currently a few other field storage systems that have been developed for Drupal 7 so far. The most cited one is probably the MongoDB module, and there is also Riak. What I am discussing in this blog post is a new field storage system for Drupal 7 which uses structWSF as the data store. This new module is called the structFieldStorage module and it is part of conStruct.

Flexibility of the Field Storage API design

The design of having one field storage system per field is really flexible and probably essential. By default, all of the field widgets and all the modules have been created using the field_sql_storage system. This means that a few things here and there have been coded with the specificities of that field storage system. The result is that even if the Field Storage API has been designed and developed such that we can create new field storage systems, the reality is that once you do it, multiple existing field widgets and modules can break from the new field storage systems.

What the field storage system developer has to do is to test all the existing (core) field widgets and modules and make sure to handle all the specifics of these widgets and modules within the field storage system. If it cannot handle a specific widget or module, it should restrict their usage and warn the user.

However, there are situations where someone may require the use of a specific field widget that won’t work with that new field storage system. Because of the flexibility of the design, we can always substitute the field_sql_storage system for the given field dependent on that special widget. Under this circumstance, the values of that field widget would be saved in the field_sql_storage system (MySQL) while the other fields would save their value in a structWSF instance. Other circumstances may also warrant this flexibility.

structFieldStorage Architecture

Here is the general architecture for the structFieldStorage module. The following schema shows how the Drupal Field Storage API Works, and shows the flexibility that resides into the fields, and how multiple fields, all part of the same bundle, can use different storage systems:

bundles_fields_field_storage_api_outline

By default, on a vanilla Drupal instance, all the fields use the field_sql_storage field storage system:

default_field_storage_system_interaction

Here is what that same bundle looks like when all fields use the structfieldstorage field storage system:

ccr_field_storage_system_interaction

Finally here is another schema that shows the interaction between Drupal’s core API, structFieldStorage and the structWSF web service endpoints:

structFieldStorage

Synchronization

Similar to the default MySQL field_sql_storage system, we have to take into account a few synchronization use cases when dealing with the structfieldstorage storage system for the Drupal content types.

Synchronization with structFieldStorage occurs when fields and field instances that use the structfieldstorage storage system get deleted from a bundle or when an RDF mapping changes. These situations don’t appear often once a portal is developed and properly configured. However, since things evolve all the time, the synchronization mechanism is always available to handle deleted content or changed schema.

The synchronization workflow answers the following questions:

  • What happens when a field get deleted in a content type?
  • What happens when a field’s RDF mapping changes for a new property?
  • What happens when a bundle’s type RDF mapping changes for a new one?

Additionally, if new field instances are being created in a bundle, no synchronization of any kind is required. Since this is a new field, there is necessarily no data for this field in the OSF, so we just wait until people start using this new field to commit new data in the OSF.

The current synchronization heuristics follow the following steps:

  1. Read the structfieldstorage_pending_opts_fields table and get all the un-executed synchronization change operations
    1. For each un-executed change:
      1. Get 20 records within the local content dataset from the Search endpoint. Filter the results to get all the entities that would be affected by the current change
        1. Do until the Search query returns 0 results
          1. For each record within that list
            1. Apply the current change to the entities
            2. Save that modified entities into the OSF using the CRUD: Update web service endpoint
      2. When the Search query returns 0 results, it means that this change got fully applied to the OSF. The state of this change record then get marked as executed.
  2. Read the structfieldstorage_pending_opts_bundles table and get all the un-executed synchronization change operations
    1. For each un-executed change:
      1. Get 20 records within the local content dataset from the Search endpoint. Filter the results to get only the ones that would be affected by the current change
        1. Do until the Search query returns 0 results
          1. For each record within that list
            1. Apply the current change to the entities
            2. Save that changed record into the OSF using the CRUD: Update web service endpoint
      2. When the Search query returns 0 results, it means that this change got fully applied to the OSF. The state of this change record then get marked as executed.

The synchronization process is triggered by a Drupal cron job. Eventually this may be changed to have a setting option that would let you use cron synchronization or to trigger it by hand using some kind of button.

Compatibility

The structFieldStorage module is already compatible with multiple field widgets and external contributed Drupal 7 modules. However, because of Drupal’s nature, other field widgets and contributed modules that are not listed in this section may be working with this new field storage system, but tests will be required by the Drupal system administrator.

Field Widgets

Here is a list of all the core Field Widgets that are normally used by Drupal users. This list tells you which field widget is fully operational or disabled with the structfieldstorage field storage system.

Note that if a field is marked as disabled, it only means that it is not currently implemented for working with this new field storage system. It may be re-enabled in the future if it become required.

Field Type Field Widget Operational?
Text Text Field Fully operational
Autocomplete for predefined suggestions Fully operational
Struct Lookup Fully operational
Struct Lookup with suggestion Fully operational
Autocomplete for existing field data Disabled
Autocomplete for existing field data and some node titles Disabled
Term Reference Autocomplete term widget (tagging) Disabled
Select list Disabled
Check boxes/radio buttons Disabled
Long text and summary Text area with a summary Fully operational
Long text Text area (multiple rows) Fully operational
List (text) Select list Fully operational
Check boxes/radio buttons Fully operational
Autocomplete for allowed values list Disabled
List (integer) Select list Fully operational
Check boxes/radio buttons Fully operational
Autocomplete for allowed values list Disabled
List (float) Select list Fully operational
Check boxes/radio buttons Fully operational
Autocomplete for allowed values list Disabled
Link Link Fully operational
Integer Text field Fully operational
Float Text field Fully operational
Image Image Fully operational
File File Fully operational
Entity Reference Select list Fully operational
Check boxes/radio buttons Fully operational
Autocomplete Fully operational
Autocomplete (Tags style) Fully operational
Decimal Text field Fully operational
Date (Unix timestamp) Text field Fully operational
Select list Fully operational
Pop-up calendar Fully operational
Date (ISO format) Text field Fully operational
Select list Fully operational
Pop-up calendar Fully operational
Date Text field Fully operational
Select list Fully operational
Pop-up calendar Fully operational
Boolean Check boxes/radio buttons Fully operational
Single on/off checkbox Fully operational

Core & Popular Modules

Revisioning

The Revisioning module is fully operational with the structfieldstorage field storage system. All the operations exposed in the UI have been handled and implemented in the hook_revisionapi() hook.

Diff

The Diff module is fully operational. Since it compares entity class instances, there is no additional Diff API implementation to do. Each time revisions get compared, then structfieldstorage_field_storage_load() gets called to load the specific entity instances. Then the comparison is done on these entity descriptions.

Taxonomy

The Taxonomy module is not currently supported by the structfieldstorage field storage system. The reason is that the Taxonomy module is relying on the design of the field_sql_storage field storage system, which means that it has been tailored to use that specific field storage system. In some places it can be used, such as with the entity reference field widget, but its core functionality, the term reference field widget, is currently disabled.

Views

structViews is a Views query plugin for querying an OSF backend. It interfaces the Views 3 UI and generates OSF Search queries for searching and filtering all the content it contains. However, Views 3 is intimately tied with the field_sql_storage field storage system, which means that Views 3 itself cannot use the structfieldstorage storage system off the shelf. However, Views 3 design has been created such that a new Views querying engine could be implemented, and used, with the Views 3 user interface. This is no different than how the Field Storage API works for example. This is exactly what structViews is, and this is exactly how we can use Views on all the fields that uses the structfieldstorage field storage system.

This is not different than what is required for the mongodb Drupal module. The mongodb Field Storage API implementation is not working with the default Views 3 functionality either, as shown by this old, and very minimal, mongodb Views 3 integration module.

structViews is already working because all of the information defined in fields that use the structfieldstorage storage system is indexed into the OSF. What structViews does is just to expose this OSF information via the Views 3 user interface. All the fields that define the local content can be added to a structViews view, all the fields can participate into filter criteria, etc.

What our design means is that the structFieldStorage module doesn’t break the Views 3 module. It does not because structViews takes care to expose that entity storage system to Views 3, via the re-implmented API.

efq_views

efq_views is another contributed module that exposes the EntityFieldQuery API to Views 3. What that means is that all of the Field Storage Systems that implement the EntityFieldQuery API should be able to interface with Views 3 via this efq_views Views 3 querying engine.

Right now, the structFieldStorage module does not implement the EntityFieldQueryAPI. However, it could implement it by implementing the hook_field_storage_query() hook. (This was not required by our current client.)

A Better Revisioning System

There is a problem with the core functionality of Drupal’s current revisioning system. The problem is that if a field or a field instance gets deleted from a bundle, then all of the values of those fields, within all of the revisions of the entities that use this bundle, get deleted at the same time.

This means that there is no way to delete a field without deleting the values of that field in existing entities revisions. This is a big issue since there is no way to keep that information, at least for archiving purposes. This is probably working that way because core Drupal developers didn’t want break the feature that enables people to revert an entity to one of its past revisions. This would have meant that data for fields that no longer existed would have to be re-created (creating its own set of issues).

However, for all the fields that uses the structfieldstorage field storage system, this issue is non-existing. Even if fields or fields instances are being deleted, all the past information about these fields remains in the revisions of the entities.

Conclusion

This blog post exposes the internal mechanism of this new structfieldstorage backend to Drupal. The next blog post will focus on the user interface of this new module. It will explain how it can be configured and used. And it will explain the different Drupal backend user interface changes that are needed to expose the new functionality related to this new module.

conStruct for Drupal 7

Frederick Giasson

construct_logo_120For more than a year we have been developing a completely new version of conStruct for Drupal 7 for one of our clients.

conStruct for Drupal 6 is really decoupled from Drupal and all the other contributed modules; in a word, it was not playing nice with Drupal. The goal of this new version has been to change that situation. The focus of this completely new conStruct module has been to create a series of connector modules that bridge most of Drupal’s core functionalities with remote structWSF instances.

We wanted to make sure that Drupal developers could manipulate content, within Drupal, that is hosted in structWSF instance(s). The best way to start aiming for that goal was to make sure that all of the core Drupal APIs commonly used by Drupal developers could be used to manipulate structWSF data like if it was native in Drupal. This is what these connectors are about.

The development of conStruct for Drupal 7 is not finished, but it is available in the Git repository. There is still refactoring and improvements required, mainly to make it easier to use and understand, but all of the code is working properly and is already used on production sites.

conStruct As a Large Scale Drupal Implementation

Those who follow the evolution of conStruct know that conStruct’s main goal is to use Drupal as a user interface for structWSF for administrative purposes, or for creating complete portals like the NOW portal. However, in our initial versions, Structured Dynamics’ purpose was to not tightly integrate with Drupal. Over time, though, we have seen broad acceptance for the Drupal front end and Drupal itself is evolving in ways compatible with semantic technologies.

What is changing with conStruct for Drupal 7, with all these connectors, is that we are now using conStruct to bridge Drupal with structWSF server instances. We supercharge Drupal 7’s capabilities with structWSF. Our evolution to a tighter Drupal coupling means the ability to manage, query, search, data mine, million of entities; to have vocabularies of tens of thousands of concepts; and to enable the querying of all of these entities and their content from any kind of devices or systems via a family of web services endpoints.

This is the initial version of what is (or should be) Drupal LSD for Structured Dynamics: A semantic web service framework backend system for Drupal.

conStruct’s Drupal Connectors

Here is the initial list of the connectors that exists:

  • structFieldStorage: this module creates a new structfieldstorage field storage system that can be used by Drupal fields to save the fields’ data into a remote structWSF instance. This is used to enable the Content Type entities to be saved into a structWSF instance. It is an extension of the Drupal field storage system
  • structEntities: this module creates a new Entity Type called the Resource Type that is used to see all the structWSF indexed records as native Entities in Drupal. This means that the Entity API can be used to manipulate any content in structWSF
  • structViews: this module creates a new data source for Views 3. This means that the Views 3 user interface is used to generate structWSF Search endpoint queries instead of SQL queries
  • structSearchAPI: this module exposes new search indexes to the Search API. This means that the Search API can be used to query a structWSF instance.

I will write about all these connectors individually in upcoming blog posts. I will cover their design, architecture and usage.

 

Neighbourhoods of Winnipeg: A Community Semantic Portal

Frederick Giasson

Introduction

I am proud to announce the new NOW (Neighbourhoods Of Winnipeg) semantic web portal! This new and innovative semantic web portal was publicly announced by the Mayor of Winnipeg City last week.

The NOW (Neighbourhoods of Winnipeg) portal is “a new Web portal (the “Portal”) produced by the City of Winnipeg to provide broad, dynamic and interactive access to local and neighbourhood information. Designed for easy access and use by all citizens, businesses, community organizations and Governments, the information on the site includes municipal data, census and demographic information, economic development information, historical data, much spatial and mapping information, and facilities for including and sharing data by external groups and constituencies.”

I would suggest you to read Mike Bergman’s blog post about this new semantic web portal to have the proper background about that initiative by the city of Winnipeg and how it uses the OSF (Open Semantic Framework) as its foundational technology stack.

This project has been the springboard that led to the Open Semantic Framework version 1.1. Multiple pieces of the framework have been developed in relation to this project, and more particularly pieces like the sWebMap semantic component and several improvements to the structWSF web services endpoints and conStruct modules for Drupal 6.

Development of the Portal

The development plan of this portal is composed of four major areas:

  1. Development of the data structure of the municipal domain by creating a series of ontologies
  2. Conversion of existing data asset using this new data structure
  3. Creation of the web portal by creating its design and by developing all the display templates
  4. Creation of new tools to let users interact with the data available on the portal

Structured Dynamics has been involved in #1, #2 and #4 by providing design and development resources, technology transfer sessions and material and supporting internal teams to create, maintain and deploy their 57 publicly available datasets.

The Data Structure

This technology stack does not have any meaning without the proper data and data structures (ontologies) in place. This gold mine of information is what drives the functionality of the portal.

The portal is driven by 12 ontologies: 2 internal and 10 external. The content of the 57 publicly available datasets is defined by the classes and properties defined in one of these ontologies.

The two internal ontologies have been created jointly by Structured Dynamics and the City of Winnipeg, but they are extended and maintained by the city only.

These ontologies are maintained using two different kind of tools:

  1. Protege
  2. structOntology

Protege is used for the big development tasks such as creating a big number of classes and properties, to do a big reorganization of the classes structure, etc.

structOntology is used for quick ontological changes to have an immediate impact on the behaviors of the portals such as label changes, SCO ontology property assignments to change the behavior of some of the tools that exist in the portal, etc.

structOntology can also be used by portal users to understand the underlying data structure used to define the data available on the portal. All users have access to the reading mode of the tool which let them browse, search and export the loaded ontologies on the portal.

The Data

Except for rare exceptions such as the historical photos, no new data has been created by the City of Winnipeg to populate this NOW portal. Most of its content comes from existing internal sources of data such as:

  • Conventional relational databases
  • GIS (Geographic Information System) on-top of relational databases
  • Spreadsheets

All of the conventional relation databases and legacy data from the GIS systems has been converted into RDF using the FME Workbench ETL system. All of the FME workbench templates are mapping the relational data into RDF using the ontologies loaded into the portal. All of the geolocated records that exist in the portal come from this ETL process and have been converted using FME.

Some smaller datasets come from internal spreadsheets that got modified to comply with the commON spreadsheet format that is used to convert spreadsheet (CSV/TSV) data files into RDF.

All of the dataset creation and maintenance is managed internally by the City of Winnipeg using one of these two data conversion and importation processes.

Here are some internal statistics of the content that is currently accessible on the NOW portal.

General Portal

These are statistics related to different functionalities of the portal.

  • Number of neighbourhoods: 236
  • Number of community areas: 14
  • Number of wards: 15
  • Number of neighbourhood clusters: 23
  • Number of major site sections: 7
  • Total number of site pages: 428,019
    • Static pages: 2,245
    • Record-oriented pages: 425,874
    • Dynamic (search-based) pages: infinite
  • Number of documents: 1,017
  • Number of images: 2,683
  • Number of search facets: 1,392
  • Number of display templates: 54
  • Number of links: 1,067
    • External links: 784
    • Internal links: 283
Site Data

These statistics show the things that are available via the portal, what are their types, their properties, what is the quantity of data that is searchable, manipulable and exportable from the portal.

  • Number of datasets: 57
  • Number of records: 425,874
    • Number of geolocational records: 418,869
      • Point of interest (POI) records: 193,272
      • Polygon records: 218,602
      • Path (route) records: 6,995
  • Number of classes (types): 84
  • Number of properties: 1,308
  • Number of triple assertions: 8,683,103

Sharing Content

An important aspect of this portal is that all of the content is contextually available, in different formats, to all of the users of the portal. Whether you are browsing content within datasets, searching for specific pieces of content, or looking at a specific record page, you always have the possibility to get your hands on the content that is being displayed to you, the user, with a choice of five different data formats:

Export Page Content

All content pages can be exported in one of the formats outlined above. In the bottom right corner of these pages you will see a Export button that you can click to get the content of that page in one of these formats.

record_export

Export Search Content

Every time you do a search on the portal, you can export the results of that search in one of the formats outlined above. You can do that by selecting the Export tab, and by selecting one of the formats you want to use for exporting the data.

browse_export

Export Datasets

You can export any publicly available dataset from the portal. These datasets have to be exported in slices if they are too big to fit in a single slice. The datasets can be exported in one of the formats mentioned above.

datasets_export

Export Census

Users also have the possibility to export census data, from the census section of the portal, in spreadsheets. They only have to select the Tables tab, and then to click the Export Spreadsheet button.

export_census

Export Ontologies

The export functionality would not be complete without the ability to consult and export the ontologies that are used to describe the content exposed by the portal. These ontologies can be read from the ontologies reader user interface, or can be exported from the portal to be read by external ontologies management tools such as Protege.

ontologies_export

Portal Design

The portal is using Drupal 6 as its CMS (Content Management System). The Drupal 6 instance communicates with structWSF using the conStruct module, which acts as a bridge between a Druapal portal and a structWSF web service network.

Here are the main design phases that have been required to create the portal:

  1. Creation of the portal’s design, and the Drupal 6 theme that implements it
  2. Creation of the Search and Browse results templates
  3. Creation of the individual records’ page design and templates based on their type
  4. Creation of the sWebMap search results templates.

The portal’s design has been created internally by the City of Winnipeg and by Tactica based on the Citizen DAN demo. Tactica also worked on another Citizen DAN like portal called MyPeg.ca.

Semantic Components

The NOW Web portal is using a series of tools that are called the Semantic Components. These are a set of Flash and JavaScript tools that can be embedded within any web page and that can easily communicate with structWSF instance(s). They display information in all kinds of charts, they can display document reading widgets, they can create dashboards of structured data, etc. The initial set of Semantic Components was developed for the MyPeg.ca project back in November 2010. This was before Steve Jobs announced that Apple would not support Adobe Flash, and far before Google announced that it would drop support for it as well.

Since the NOW portal wanted to re-use as much as possible to lower the development cost related to the portal, they choose to use the complete OSF stack which includes these Semantic Components.

However, when we participated in developing this new NOW portal, we did extended the set of Semantic Components by creating the most complex Semantic Component: the sWebMap. However, because of the two announcements mentioned above, we choose to move forward and to create the sWebMap Semantic Component using JavaScript instead of Flash. The other Semantic Component tools that have been developed in Flash have not yet been ported into JavaScript.

Conclusion

The new NOW semantic web portal’s main asset is its data: how it can be searched (with traditional search engines or using a semantic component to search, browse, filter and localize results), displayed and exported. This portal has been developed using a completely free and open source semantic platform that has been developed from previous projects that open sourced their code.

I consider this portal a pioneer in the way municipal organization will provide new online services to their citizens and to the commercial enterprises based on the quality of the data that will be exposed via such Web portals.

Building conStruct Templates

Frederick Giasson
The templating engine in conStructis one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

  • Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.
  • Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.
  • Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on “text”, you will get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.


As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

 

Building conStruct Templates

The templating engine in conStruct is one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…

Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

�Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.

�Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.

�Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on �text�, you will

Building conStruct Templates

The templating engine in conStruct is one of the older modules that has been developed. In fact, it was probably more than two years ago. However, it has never been documented…

Until now!

It is nice to have a framework (structWSF) that lets you ingest, manage, analyze and publish structured, semi-structured and unstructured data using a set of web service endpoints. However, it would be even nicer to have a user interface to interact with that system. It is what conStruct is: a user interface to structWSF endpoints.

But it would be even nicer to have a way to personalize the structWSF data presented to users via conStruct: it is what the conStruct templates are all about.

The only remaining problem was for someone new to conStruct to know how to create these templates.

Quick Introduction to conStruct Templates

conStruct templates are used to create different HTML page layouts to present information about instance records depending on their type. The present design utilizes Smarty as the baseline templating framework to create these templates. The only thing we added to it is a new API that lets template designers have access to the instance record(s) description(s). Think of it as an API to manipulate the information that will be displayed in the HTML page.

The general idea is that depending on the type of the instance record you want to display information about, you will want your system to display the information you have about that instance record differently depending on the type of that instance record. For example, if you want to present information about a neighborhood, the information will be displayed differently than if you want to present information about a person.

By using Smarty and the API extension created for conStruct, template designers are able to create different layouts to display things differently depending on the type of the records being viewed by the users of the system. Designers will also be able to make their template adaptive depending on how the records are described. As we know, not all persons are described the same way!

Once a template designer gets used to this system, these presentation templates for instance records become an effective way to present information to users, in all different ways, by leveraging the structure of the information it conveys.

Some Examples

Here are some examples of templated pages for different kind of instances. These should show you how different they can be, and how they can integrate external tools such as Flash movies, YouTube videos and uses external JavaScript libraries:

  • Johnson County, Iowa (Citizen Dan). This template displays the information about a County. It displays some textual information such as the total population, etc. It also embeds a map, and other Semantic Components tools.
  • Sargent Park (MyPeg.ca). This template is like the one above, but it displays information about a Neighborhood of Winnipeg City. It displays a map as well, but it also displays a Dashboard session.
  • Lonny & Greta (MyPeg.ca). This template is quite different. It displays information about an Interview. It embeds a series of videos from YouTube. And if you click on �text�, you will get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.

As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

get a text summary of the interview that is displayed in the sStory semantic component that got created using jQuery.

As you can see, you have the leisure to do whatever you want by leveraging this templating system. And I am not even talking about the other goodies that you could get from it if you start leveraging the Drupal API within these templates!

Yes, But How?

The last question that I have to answer with this blog post is: how the hell can I create these templates?

The answer to this question (I hope) lies in this new TechWiki page that I published this morning:

Building conStruct Templates

It explains everything you need to know in order to create a conStruct template: what is the workflow, how the system works internally, a complete description of the extension API with a lot of Smarty code examples.

Enjoy!

New versions of structWSF and conStruct

Frederick Giasson

triple_120construct_logo_120

We just released a new (major) version of both structWSF and conStruct. Though some months had passed since we last released this software, we finally got the time and opportunity to make these important upgrades.Many things have changed in both packages. I don’t want to iterate all the changes in this blog post, so I would suggest you to read the changes log files here:

These new versions have greatly been impacted by the needs of our clients. We also started to introduce some new concepts we wrote about the last few months.

A really good addition to this release is the a brand new Installation Manual. Hopefully people will be able to “easily” and properly install and setup a Web server to host these two packages.

All documentation files have been updated:

You can download both software packages from here:

An Amazon EC2/EBS Architecture

Some of the changes to these new versions have been made to help create, setup and maintain Web servers that host structWSF and conStruct instances.

At Structured Dynamics, we have developed and use a server architecture that leverages Amazon computer-in-the-clouds services such as: EC2, EBS, Elastic IP in the Cloud. Such an architecture is giving us the flexibility to easily maintain and upgrade server instances, to instantly create new structWSF instances in one click (without performing all these steps everytime), etc.

You can contact us for more information about these EC2 AMIs and EBS Volumes that we developed for this purpose. Here is an overview of the architecture that is now in place:

structwsf_amazon

There is a clear separation of concerns between three major things:

  • Software & libraries
  • Configuration files
  • Data files.

We chose to put all software and libraries needed to create a stand-alone structWSF instance in an EC2 AMI. This means that all needed software to run a structWSF instance is present on the Virtuoso server running Ubuntu server.

Then we chose to put all configuration and data files on an EBS volume that we attach, and mount, on the EC2 instance. You can think about a EBS volume as a physical hard drive: it can be mounted on a server instance, but it can’t be shared between multiple instances.

By splitting the software & libraries, configuration and data files, we make sure that we can easily upgrade a structWSF server in production with the latest version of structWSF (its code base and all related software such as Virtuoso, Solr, etc). Since the configuration and data files are not on the EC2 instance, we can easily create a new EC2 instance by using the latest structWSF AMI we produced, and then to mount the configuration and data files EBS volume on the new (and upgraded) structWSF instance. That way, in a few clicks, we can fully upgrade a server in production without fear of disturbing the configuration or data files.

Additionally, we can easily create backups of configuration and data files at different intervals by using Amazon’s Snapshot technology.

Finally, we chose to put all related software and configuration files needed to run a conStruct instance in another, separate, EBS volume. That way, we have a clean structWSF AMI instance that can be upgraded at any time, and we can plug (mount) a conStruct instance (EBS instance) into a structWSF server at any time. This means that we can easily have structWSF instances with or without a conStruct instance. The same strategy can easily be used to create plugin packages that can be mounted and unmounted to any structWSF instance at any time, depending on the needs.

All this makes structWSF server instances maintenance easier, simpler and faster.