RDF Aggregates and Full Text Search on Steroids with Solr

Frederick Giasson

Preamble

As I explained in my latest blog post, I am now starting to talk about a couple of things I have been working on in the last few months that will lead to a release, by Structured Dynamics, in the coming months. This blog post is the first step into that path. Enjoy!

Introduction

I have been working with RDF, SPARQL and triple stores for years now. I have created many prototypes and online services using these technologies. Having the possibility to describe everything with RDF, and having the possibility to index everything in a triple store that you can easily query the way you want using SPARQL, is priceless. Using RDF saves development and maintenance cost because of the flexibility of store (triple store), the query language (SPARQL), and associated schemas (ontologies).

However, even if this set of technologies can do everything, quickly and efficiently, it is not necessarily optimal for all tasks you have to do. As we will see in this blog post, we use RDF for describing, integrating and managing any kind of data (structured or unstructured) that exists out there. RDF + Ontologies are what we use as the canonical expression of any kind of data. It is the triple store that we use to aggregate, index and manage that data, from one or multiple data sources. It is the same triple store that we use to feed any other system that can be used in our architecture. The triple store is the data orchestrator in any such architecture.

In this blog post I will show you how this orchestrator can be used to create Solr indexes that are used in the architecture to perform three functions that Solr has been built to perform optimally: full-text search, aggregates and filtering. So, while a triple store can perform these functions, it is not optimal for what we have to do.

Overview

The idea is to use the RDF data model and a triples store to populate the Solr schema index. We leverage the powerful and flexible data representation framework (RDF), in conjunction with the piece of software that lets you do whatever you want with that data (Virtuoso), to feed a carefully tailored Solr schema index to optimally perform three things: full-text search, aggregates and filtering. Also, we want to leverage the ontologies used to describe this data to be able to infer things vis-à-vis these indexed resources in Solr. This leverage enables us to use inference on full-text search, aggregates and filtering, in Solr! This is quite important since you will be able to perform full text searches, filtered by types that are inferred!

Some people will tell me that they can do this with a traditional relational database management system: yes. However, RDF + SPARQL + Triple Store is so powerful to integrate any kind of data, from any data sources; it is so flexible that it saves precious development and maintenance resources: so money.

Solr

What we want to do is to create some kind of “RDF” Solr index. We want to be able to perform full-text searches on RDF literals; we want to be able to aggregate RDF resources by the properties that describe them, and their types; and finally we want to be able to do all the searches, aggregation and filtering using inference.

So the first step is to create the proper Solr schema that will let you do all these wonderful things.

The current Solr index schema can be downloaded here. (View source if simply clicking with your browser.)

Now, let’s discuss this schema.

Solr Index Schema

A Solr schema is composed of basically two things: fields and type of fields. For this schema, we only need two types of fields: string and text. If you want more information about these two types, I would refer you to the Solr documentation for a complete explanation of how they work. For now, just consider them as strings and texts.

What interests us is the list of defined fields of this schema (again, see download):

  • uri [1] – Unique resource identifier of the record
  • type [1-N] – Type of the record
  • inferred_type [0-N] – Inferred type of the record
  • property [0-N] – Property identifier used to describe the resource and that has a literal as object
  • text [0-N] (same number as property) – Text of the literal of the property
  • object_property [0-N] – Property identifier used to describe the resource where the object is a reference to another resource and that this other resource can be described by a literal
  • object_label [0-N] (same number as object_property) – Text used to refer to the resource referenced by the object_property

Full Text Search

A RDF document is a set of multiple triples describing one or multiple resources. Saying that you are doing full-text searches on RDF documents is certainly not the same thing as saying that you are doing full-text searches on traditional text documents. When you describe a resource, you rarely have more than a couple of strings, with a couple of words each. It is generally the name of the entity, or a label that refers to it. You will have different numbers, and sometimes some description (a short biography, or definition, or summary, as examples). However, except if you index an entire text document, the “textual abundance” is quite poor compared to an indexed corpus of documents.

In any case, this doesn’t mean that there are no advantages in doing full-text searches on RDF documents (so, on RDF resource descriptions). But, if we are going to do so, let’s do so completely, and in a way that meets users’ expectations for full-text document search.  By applying this mindset, we can apply some cool new tricks!

Intuitively the first implementation of a full-text search index on RDF documents would simply make a key-value pair assignment between a resource URI and its related literals. So, when you perform a full-text search for “Bob”, you get a reference on all the resources that have “Bob” in one of the literals that describe these resources.

This is good, but this is not enough. This is not enough because this breaks the more basic behavior for any users that uses full-text search engines.

Let’s say that I know the author of many articles is named “Bob Carron”. I have no idea what are the titles of the articles he wrote, so I want to search for them. With the system exposed above, if I do a search for “Bob Carron”, I will most likely get back as a result the reference to “Bob Carron”, the author person. This is good, but this is not enough.

On the results page, I want the list of all articles that Bob wrote! Because of the nature of RDF, I don’t have this “full-text” information of “Bob” in the description of the articles he wrote. Most likely, in RDF, Bob will be related to the articles he wrote by reference (object reference with the URIs of these articles), i.e., <this-article> <author> <bob-uri>. As you can notice, we won’t get back any articles in the resultset for the full-text query “Bob Carron” because this textual information doesn’t exist in the index at the level of the articles he wrote!

So, what can we do?

A simple trick will beautifully do the work. When we create the Solr index, what we want is to add the textual information of the resources being referenced by the indexed resources. For example, when we create the Solr document that describes one of the articles written by Bob, we want to add the literal that refers to the resource(s) referenced by this article. In this case, we want to add the name of the author(s) in the full-text record of that article. So, with this simple enhancement, if we do a search for “Bob Carron”, we will now get the list of all resources that refers to Bob too! (articles he wrote, other people that know him, etc).

So, this is the goal of the “object_property” and “object_label” fields of the Solr index. In the schema above, the “object_property” would be “author” and the “object_label” would be “Bob Carron”. This information would belong to the Solr document of the Article 1.

Full Text Search Prototype

Let’s take a look at the prototype running system (see screen capture below).



The dataset loaded in this prototype is Mike’s Sweet Tools. As you notice in the prototype screen, many things can be done with the simple Solr schema we published above. Let’s start with a search for the word “test”. First, we are getting a resultset of 17 things that have the “test” word in any of their text-indexed fields.

What is interesting with that list is the additional information we now have for each of these resultsets that come from the RDF description of these things, and the ontologies that have been used to describe them.

For example, if we take a look at Result #4, we see that the word “test” has been found in the description of the Ontology project for the “TONES  Ontology Repository” record. Isn’t that precision far more useful than saying: the word “test” has been found in “this webpage”? I’ll let you think about it.

Also, if we take a look at Result #1, we know that the word “test” has been found in the homepage of the Data Converter Project for the”Talis Semantic Converter” record.

Additionally, by leveraging this Solr index, we can do efficient aggregates on the types of the things returned in the resultset for further filtering. So, in the section “Filter by kinds” we know what kinds of things are returned for the query “test” against this dataset.

Finally, we can use the drop-down box at the right to do a new search (see screenshot), based on the specific kind of things indexed in the system. So, I could want to make a new search, only for “Data specification projects” with the keyword “rdf”. I already know from the user interface that there are 59 such projects.

All this information comes form the Solr index at query time, and basically for free by virtue of how we set up the system. Everything is dynamically aggregated and displayed to the user.

However, there are a few things that you won’t notice here that are used:  1) SPARQL queries to the triple store to get some more information to display on that page; 2) the use of inference (more about it below), and; 3) the leveraging of the ontologies descriptions.

In any case, on one of SD’s test datasets of about 3 million resources, such a page is generated within a few hundred milliseconds: resultset, aggregates, inference and description of things displayed on that page.  This same 3 million resources that returns results in a few hundred milliseconds did so on a small Amazon EC2 server instance for 10 cents per hour. How’s that for performance?!

Aggregates and Filtering on Properties and Types

But, we don’t want to merely do full-text search on RDF data. We also want to do aggregates (how many records has this type, or this property, etc.) and filtering, at query time, in a couple of milliseconds. We already had a look at these two functions in the context of a full-text search. Now let’s see it in action in some dataset prototype browsing tools that uses the same Sweet Tools dataset.

In a few milliseconds, we get the list of different kind of things that are indexed in a given dataset. We can know what are the types, and what is the count for each of these types. So, the ontologies drive the taxonomic display of the list of things indexed in the dataset, and Solr drives the aggregation counts for each of these types of things.

Additionally, the ontologies and the Virtuoso inference rules engine are used to make the count, by inference. If we take the example of the type “RDF project”, we know there are 49 such projects. However, not all these projects are explicitly typed with the “RDF project” type. In fact, 7 of these “RDF project” are “RDF editor project” and 6 are “RDF generator project”.

This is where inference can play an important role: an article is a document. If I browse documents, I want to include articles as well. This “broad context retrieval” is driven by the description of the ontologies, and by inference; this is the same thing for these projects; and this is the same thing for everything else that is stored as structured RDF and characterized by an ontology.

The screenshot above shows how these inferences and their nestings could present themselves in a user interface.

Once the user clicks on one of these types, he starts to browse all things of that type. On the next screenshot below, Solr is used to add filters based on the attributes used to describe these things.

In some cases, I may want to see all the Projects that have a review. To do so, I would simply add this filter criteria on the browsing page and display the “Projects” that have a “review” of them. And thanks to Solr, I already know how many such Projects have reviews, right before even taking a look at them.

Note, then, on this screenshot that the filters and counts come from Solr.  The list of the actual items returned in the resultset comes from a SPARQL query, and the name of the types and properties (and their descriptions) come from the description of the ontologies used.

This is what all this stuff is about: creating a symbiotic environment where all these wonderful systems live together to do the effective management of the structured data.

Populating the Solr Index

Now that we know how to use Solr to perform full-text searches, and the aggregating and filtering of structured data, one question still remains: how do we populate this index? As stated at above, the goal is to manage all the structured data of the system using a triple store and ontologies. Then it is to use this triple store to populate the Solr index.

Structured Dynamics uses the Virtuoso Open Source as the triple store to populate this index for multiple reasons. One of the main ones is for its performance and its capability to do efficient basic inference. The goal is to send the proper SPARQL queries to get the structured data that we will index in the Solr schema index that we talked about above. Once this is done, all the things that I talked about in this blog post become possible, and efficient.

Syncing the Index

However, in such a setup, we have to keep one thing in mind: each time the triple store is updated (a resource is created, deleted or updated), we have to sync the Solr index according to these modifications.

What we have to do is to detect any change in the triple store, and to reflect this change into the Solr index. What we have to do is to re-create the entire Solr document (the resource that changed in the triple store) using the <add /> operation.

This design raises an issue with using Solr: we cannot simply modify one field of a record. We have to re-index the entire description of the document even if we want to modify a single field of any document. This is a limitation of Solr that is currently addressed in this new feature proposition; but it is not currently available for prime time.

Another thing to consider here is to properly sync the Solr index with any ontology changes (at the level of the class description) if you are using the inference feature. For example, assume you have an ontology that says that class A is a sub-class-of class B. Then, assume the ontology is refined to say that class A is now a sub-class-of class C, which itself is a sub-class-of class B. To keep the Solr index synced with the triple store, you will have to perform all modifications that affect all the records of these types. This means that the synchronization doesn’t only occur at the level of the description of a record; but also at the level of the changes in the ontologies used to describe those records.

Conclusion

One of the main things to keep in mind here is that now, when we develop Web applications, we are not necessarily talking about a single software application, but a group of software applications that compose an architecture to deliver a service(s). In any such architecture, what is at the center of it is Data.

Describing, managing, leveraging and publishing this data is at the center of any Web service. It is why it is so important to have the right flexible data model (RDF), with the right flexible query language (SPARQL), and the right data management system (triple store) in place. From there, you can use the right tools to make it available on the Web to your users.

The right data management system is what should be used to feed any other specific systems that compose the architecture of a Web service. This is what we demonstrated with Solr; but it is certainly not limited to it.

Different World Views (TBox) for the same Structs (ABox)

Frederick Giasson

Mike continues his series of blog posts that talks about the distinction between ABoxes (the assertions box; the data instances box) and TBoxes (the terminologies box; the data schemas box). Mike suggests to people to make a distinction between the data instances (individuals) that belongs to the ABox, and the vocabularies (schemas, ontologies; whatever how you call these formal specifications of conceptualizations) that belongs to the TBox.

I wanted to hammer an important point that emerged in our recent discussions about these specific questions: the TBox defines the language used to describe different kind of things and the ABox is the actual description of these things. However, there is an important distinction to make here: there is a difference between using some properties to describe a thing and understanding the meaning of the use of these properties to describe these things.

Let’s take the use-case of two systems that exchange data. The data instances that will be transmitted between the two systems will be exactly the same: their ABox description will be the same; they will use the same properties and the same values to describe the same things. However, nothing tells us how each of these properties will be processed, understood and managed by these two systems. Each system has its own Worldview. This mean that their TBox (the meaning of classes and properties used to describe data instances) will probably be different, and so, interpreted and handled differently.

I think the fact that two systems may process the same information differently is the lesser evil. This is no different than how humans communicate. Different people have different Worldviews that will dictate how they will see and reason over things. One person can see a book and think at it as a piece of art where another person can say: “Great! I finally have something to start that damned fire!”. The description of the thing (the book) didn’t change; but its meaning changed from one person to another. Exactly the same thing applies to systems that are exchanging data instances.

This is really important since considerations of the TBox (how data instances are interpreted) shouldn’t be bound to the considerations of the ABox (the actual data instances that are transmitted). Otherwise no systems will ever be able to exchange data considering that they will most than likely always share different Worldviews for the same data (they will handle and reason over data instances differently).

I think this is a really important thing to keep in mind going forward because there won’t ever be a single set of ontologies to describe everything on the semantic web. There will be multiple ontologies that will describe the same things, and there will be an endless number of versions of these ontologies (there are already many). And finally, the cherry on the cake, how these ontologies are handled and implemented in systems is different!

But take care here; this doesn’t mean that we can’t exchange meaningful data between different systems. This only means that different Worldviews exist, which means that care should also be given to not mix data with the interpretation of concepts.  This is yet another  reason why we have to split apart concerns between the ABoxes and the TBoxes.

The Next Bibliographic Ontology: OWL

Frederick Giasson

The Bibliographic Ontology‘s aim is to be expressive and flexible enough to be able to convert any existing bibliographic legacy schema (such as Bibtex and its extensions, MARC, Elsevier’s SDOS & CITADEL citation schemas, etc.) and RDFS/OWL ontologies to it.

This new BIBO version 1.2 is the result of more than one year of thinking and discussions between 101 community members and 1254 mail messages. The project’s first aim of expressiveness and flexibility is nearly reached. BIBO’s ongoing development is now pointing to a series of methods and best practices for mature ontology development.

Some BIBO mappings between legacy schemas have been developed, but this trend will now be accelerated. More people are getting interested in BIBO’s ability to describe bibliographic resources. Some people are interested in it to describe bibliographic citations; others are interested in it to integrate data from different bibliographic data sources, using different schemas, into a single and normalized data source. This single data source (in RDF) can then become easily queried, managed and published. Finally, other people are interested in it as a standard agreed to by an open community, that helps them to describe bibliographic data that aims to be published and consumed by different kind of data consumers (such as standalone software like Zotero; or such as citation aggregation Web services like Scirus or Connotea).

With this BIBO 1.2 release, much has changed and been improved. Now, it is time for the community to start implementing BIBO in different systems; to create more mappings; and to complete more converters.

Design Redux

As you may recall from its early definition, BIBO has been designed for both: (1) a core system with extensions relevant to specific domains and uses, and (2) a collaborative development environment governed by the community process.

These design imperatives have guided much of what we have done in this new version 1.2 release to aid these objectives.

BIBO in OWL 2

The new version of BIBO is now described using OWL 2. In the next sections you will know why we choose to use OWL 2 as the way to describe BIBO in the future. However, saying that it is OWL 2 doesn’t mean that it becomes incompatible with everything else that exists. In fact, it validates OWL 1.1 and its DL expressivity is SHOIN(D); this means that fundamentally nothing has changed, but that we are now leveraging a couple of new tools and concepts that are introduced by OWL 2.

As you will see below this decision results in much more than a single update of the ontology. We are introducing an updated, and more efficient, architecture to develop open source ontologies such as The Bibliographic Ontology.

New Versioning System

OWL 2 is introducing a new versioning and importation system for OWL ontologies. This feature alone strongly argued for the adoption of OWL 2 as the way to develop BIBO in the future.

This new versioning system consists of two things: an ontologyURI and a versionURI. The heuristics to define, check, and cache an ontology that as an ontologyURI and possibly a versionURI are described here.

BIBO has an ontologyURI and multiple versionURIs such as http://purl.org/ontology/bibo/1.0/, http://purl.org/ontology/bibo/1.1/, and http://purl.org/ontology/bibo/1.2/.

Right now, the current version of the ontology is 1.2. This means that the current version of BIBO will be located at two places: http://purl.org/ontology/bibo/ and http://purl.org/ontology/bibo/1.2/.

The location logic of ontologies is described here. What we have to take care here is that if someone dereferences any class or properties of BIBO, it will always get the description of that class or property from the latest version of the ontology. This is why the caching logic is quite important. The user agent has to make sure that it caches the version of the ontology that it knows.

What is really important to understand is that the URI of the ontology won’t change over time when we introduce new versions of the same ontology. Only the location of these versions will change.

Finally, the OWL 2 mapping to RDF document tells us that we have to use the owl:versionInfo OWL property to define the versionURI of an ontology. This is the reason why the use of this OWL 2 versioning system doesn’t affect the validity of BIBO as a OWL 1.1 ontology; because owl:versionInfo is also an OWL 1.1 property.

Now, lets take look at the tools that we will use to continue the development of BIBO.

Protégé 4 for Developing BIBO

We chose to now rely on Protégé 4 to develop BIBO in the future. We wanted to start using a tool that would help the community to develop the ontology. Considering that Protégé 4 Beta has been released in August; that it supports OWL 2 by using the OWLAPI library; and many plugins are already supported; it makes it the best free and open-source option available.

What I have done is to add some SKOS annotation properties to annotate the BIBO classes and properties to help us to edit and comment on the ontology. Here is the list of new annotation properties we introduced:

  • skos:note, is used to write a general notes
  • skos:historyNote, is used to write some historical comments
  • skos:scopeNote, is really important. It is the new way to target the classes and properties, imported from external ontologies, that we recommend to use to describe one aspect of BIBO. The scopeNote will tell the users the expected usage for these external resources.
  • skos:example, is used to give some examples that show how to use a given class or property. Think of RDF/XML or RDF/N3 code examples.

Finally, all these annotations are included in BIBO’s namespace.

OWLDoc for Generating Documentation

OWLDoc is a plugin for Protégé that generates documentation for OWL ontologies. In a single click, we can now get the complete documentation of an ontology. This makes the generation of the documentation for an ontology much, much, more efficient. Users can easily see which ontologies are imported, and then they can easily browse the structure of the ontology. Many facets of the ontology can be explored: all the imported ontologies, the classes, the object/data properties, the individuals, etc.

You can have a look at the new documentation page for BIBO here. On the top-left corner you have a list of all imported ontologies. Then you can click on facet links to display related classes, properties or individuals. Then you may read the description of each of these resources, their usage, and their annotations (scope-notes, Etc.).

Please note there are still some issues and improvements to do with the template used to generate the pages, such as multiple resource descriptions not yet adequately distinguished. We are in the process of cleaning up these minor issues. But, all-in-all, this is a major update to the workflow since any user can easily re-create the documentation pages.

Collaborative Protégé for Community Development

Now that it is available for Protégé 4, we will shortly setup a Protégé server and make it available to the community to support BIBO’s community development. We will shortly announce the availability of this Collaborative Protégé.

In the meantime, I suggest to use the file “bibo.xml” from the “trunk” branch of the SVN repository (see Google Code below). The Bibliographic Ontology can easily be opened that way using the “Open…” option to open the local file of the SVN folder, or by using the “Open URI…” option to open the bibo.xml file from the Google Code servers. That way, each modification to the ontology can easily be committed to the SVN instance.

Google Code to Track Development

As noted above, the BIBO Google Code SVN is used to keep track of the evolution of the ontology. All modifications are tracked and can easily be recovered. This is probably one of the most important features for such a collaborative ontology development effort.

But this is not the only use of this SVN repository. In fact, it as an even more central role: it is the SVN repository that sends the description of the ontology for any location query, by any user, for any version. Below we will see the workflow of a user query that leads the SVN repository to send back a description for the ontology.

Google Groups to Discuss Changes

The best tool to discuss ontology development is certainly a mailing list. A Google Groups is an easy way to create and manage an ontology development mailing list. It is also a good way to archive and search discussions that has an impact on the development (and the history) of the ontology.

Purl.org to Access the Ontology

Another important piece of the puzzle is to have a permanent URI for an ontology that is hosted by an independent organization. That way, even if anything happens with the ontology development group, hopefully, the URI will remain the same over time.

This is what Purl.org is about. It adds one more step to the querying workflow (as you will notice in the querying schema bellow), but this additional step is worth it.

General Query Workflow

There is one remaining thing that I have to talk about: the general querying workflow. I have been talking about the new OWL 2 versioning system, purl.org redirection and using the SVN repository to deliver ontology descriptions. So, there is what the workflow looks like:

[clik to enlarge this schema]

At the first step, the user requests the rdf+xml http://purl.org/ontology/bibo/. As we discussed above, this permanent URI is hosted by Purl.org; what this service does is to redirect the user to the location of the content negotiation script.

At the second step, the user requests the rdf+xml serialization of the description of the ontology at the URI of the location sent by the Purl.org server: http://conneg.com/script/. One of the challenges we have with this architecture is that neither Purl.org nor Google Code handles content negotiation with a user.

Thus, it is also necessary to create a “middle-man” content negotiation script that performs the content negotiation with the user, and redirects it to the proper file hosted on SVN repository. (If Purl.org or the SVN repository could handle the content negotiation part of the workflow, we could then remove the step #2 from the schema above and then improve the general architecture.  However, for the present, this step is necessary.)

Note 1: Take a special look at the redirection location sent back by the content negotiation script: http://…/tags/1.2/bibo.xml. This is a direct cause of the new versioning has the versionURI http://purl.org/ontology/bibo/1.2/. Considering the versioning system, the content negotiation script redirects the user to the description of the latest version of the ontologyURI (which is currently the version 1.2).

Note 2: Purl.org current doesn’t strictly conform with the TAG resolution on httpRange-14. However this should be resolved in an upgrade of the Purl.org system that is underway (the current system is dated as of the early 1990s).

At the third step, the SVN repository returns the requested document by the user with the proper Content-Type.

Conclusion

Developing open source ontologies is not an easy task. Development is made difficult considering the complexity of some ontologies, considering the different way to describe the same thing and considering the level of community involvement needed.  Thus, open source ontology development needs the proper development architecture to succeed.

I have had the good fortune to work on the this kind of ontology development with Yves Raimond on the Music Ontology, with Bruce D’Arcus on the Bibliographic Ontology, and with Mike Bergman on UMBEL. Each of these projects has led to an improvement of this architecture. After two years, these are the latest tools and methods I can now personally recommend to use to collectively create, develop and maintain ontologies.

UMBEL Web Services Endpoints Released

Frederick Giasson
After some delay, we are pleased to finally release the UMBEL Web services endpoints to the public. We have re-organized the Web services we introduced three months ago to add coherency and flexibility to the model.

The goal remains the same, but with a different flavor: these tools let ontologists and Web developers search, discover and use the UMBEL subject concept and named entity structures. The added flavor is that these Web services now fully embrace the HTTP 1.1 protocol and are provided via a series of well established data and serialization formats.

We now have RESTful Web services to add to our RESTful linked data. Pretty cool combination!

We are introducing two kinds of Web services: (1) atomic Web services and (2) compound Web services. An atomic Web service only performs one action: It takes some inputs and then outputs a resultset of the action. A compound Web service takes multiple atomic Web services, plugs them together in a pipeline model, and then takes some inputs and outputs a resultset arising from the compound action.

The communication between each of these Web service instances and the external World is the same: communication is governed by the HTTP 1.1 protocol. HTTP is generally RESTful and used to establish the communication, to determine mime type and serialization, to get inputs, to return status of the communication and possible errors, and to send back the resultset of the computation of the Web service.

That way, we can easily, within hours, programmatically pipeline these atomic Web services together to create new Web services. We can integrate external Web services endpoints into the same pipeline without modifying anything to the architecture. Status, errors and resultsets are propagated along the line, directly to the data consumer. This is the flexibility part of the story.

Now, how cool is that?

Overview of the UMBEL Web Services Endpoints

We are today releasing a couple of these atomic and compound Web service endpoints to the public, but others will follow in the coming weeks and months. Four families of Web services have been released that total seven Web service endpoints:

If you don’t know what UMBEL is, I would suggest you read a background information page that talks about the project.

The most important reading related to this blog post is the API philosophy documentation page that talks about the details of the design of this Web services architecture.

For Web developers that want to integrate these Web services endpoints within their application, an API documentation page explains how to communicate with these endpoints for each of the services.

Example of an Atomic Web Service

The Inference: Lister Web service is a good example of an atomic Web service. It takes a subject concept URI as the input and outputs a series of super-class-of, sub-class-of or equivalent-class-of classes for that concept. As an atomic service it does one thing and one thing only: Inferring relationships of a given subject concept URI.

Example of a Compound Web Service

The Reporter: Named Entity Web service is a good example of a compound Web service. This Web service displays full of information about a UMBEL named entity URI. However, not all the information returned by this Web service is directly computed by it. In fact, the information about broader and equivalent classes and subject concepts come from the Inference: Lister Web service. Results coming from this Web service are immediately integrated in the Reporter’s resultset. This is easily done considering that they share the same communication language (HTTP 1.1) and the same data and serialization formats (XML, RDF+XML and RDF+N3). This flexibility is priceless to quickly create resourceful compound Web services.

Conclusion

After some months to get the design right, we have finally released some of the UMBEL Web services to the public. These Web services can easily be integrated in current software architectures to leverage UMBEL’s vision of the World. The architecture underlying what we have released today will help to easily integrate UMBEL’s principles and concepts within new and existing projects. This will ultimately help people to quickly react to the changing World of needs and expectations of data users and consumers.

I hope you will enjoy using these new Web services, which Zitgist is freely hosting. The data you get from the Web service is open data and can be used freely with attribution.

Please do report any issues you may encounter. We also welcome any advice or suggestions that you would care to provide to enhance the overall system.

Exploding DBpedia’s Domain using UMBEL

Frederick Giasson

A couple of challenges I have found with DBpedia is that it is hard for a system to interact with the dataset and it is hard to figure out how to interpret information instantiated in it. It is hard to know what properties are used to describe individuals; and hard to know what the classes refer to. It is also hard for standalone and agent software to understand the nature of the individuals that are instantiated by DBpedia because the classes they belong to are generally unknown or poorly defined.

In the following blog post I suggest to use a method known as “exploding the domain” to try to overcome these difficulties of using and understanding DBpedia. This adds still further usefulness to DBpedia’s considerable value. This demonstration is based on the UMBEL subject concept structure.

As I will demonstrate below, this method consists of contextualizing classes in a coherent framework to explode their domains. By exploding the domain of a class, we link it to other classes that are defined by external ontologies. By exploding the domain of a class by linking it to externally defined classes, we also help standalone and agent software to understand the meaning for that class (at least if they understand the meaning of the classes that have been linked to it). Note that we are able to explode the domains by linking classes using only three properties: rdfs:subClassOf, owl:equivalentClass and umbel:isAligned.

First of all, let me give some background information about how DBpedia individuals and UMBEL named entities have been created, and how both datasets have been linked together.

How DBpedia individuals are instantiated

DBpedia is a dataset that is based on the well known Wikipedia encyclopedia. Basically DBpedia creates one individual for each Wikipedia page. Most of the individuals that are instantiated in this way are what we call a “named entity” in UMBEL’s parlance.

But to be instantiated, an individual has to belong to a class. DBpedia chooses to use Yago‘s classification system (that is based on WordNet) to instantiate those DBpedia individuals. This means that all DBpedia individuals belong to at least (theoretically) one Yago class. This means that all DBpedia individuals are instances of Yago classes (and in some rarer cases, they are also instances of classes defined in external ontologies).

How UMBEL named entities have been created

For its part, UMBEL’s named entities dictionaries come from different data sources. Currently, most all public UMBEL named entities also come from Yago (example: Aristotle), but many also come from the DBTune dataset (example: Pete Baron) or others. (UMBEL’s design allows more named entities to be plugged into the system as additional dictionaries at will.)

However, unlike DBpedia, we do not use Yago’s classification system to instantiate these named entities. And unlike Yago, we do not use the WordNet classes to instantiate the named entities either.

The current UMBEL subject concept structure is based on OpenCyc. This means that the relations between the classes that instantiate the UMBEL named entities come from the Cyc knowledge base.

So while we use Yago’s named entities (from Wikipedia) as a starting basis, we instantiate them using the UMBEL subject concept classes instead of the WordNet classes. So, basically, we have switched the WordNet conceptual framework for the UMBEL (or OpenCyc) one.

But, how did we create these UMBEL named entities, instantiated using UMBEL subject concept classes and based on Yago? Here is the linkage path:

Yago classes –> WordNet synsets <– Cyc collections <– OpenCyc classes <– UMBEL subject concept classes

Et voilà !

How UMBEL named entities are linked to DBpedia individuals

OK, so now how do we link UMBEL named entities to DBpedia individuals? It is simple. Remember that DBpedia individuals have been created from Wikipedia pages. Also remember that Yago individuals come from the same Wikipedia pages. We can then make the link between the individuals from DBpedia and the individuals from Yago based on Wikipedia URLs.

Exactly the same logic applies for linking DBpedia individuals to UMBEL named entities.

The end result of this linkage is that we have UMBEL named entities that are the same as DBpedia individuals. The difference is that the UMBEL named entities are now instances of UMBEL subject concepts: a totally different conceptual structure.

Remember that these named entities are contextualized in a coherent conceptual framework. And this characteristic means a lot for what is yet to come.

Web services to search and visualize these named entities

We created two new web services on the UMBEL web services home page (the user interface to these web services; the endpoints will be released later) to help people interact with these named entities:

  1. The “Search Named Entities Dictionaries” web service
  2. The “Named Entity Detailed Report” web service

The first web service lets you search amongst all publicly available UMBEL named entities dictionaries.

The second web service lets you visualize detailed information about any named entity.

This information page shows you the full scope of information about a named entity: which class it belongs to (subject concept classes as well as external classes); which other individuals, from other datasets, are identical to them; examples of web services that get queried with information about this named entity; etc.

Exploding the domain of Plato

Now that this background information has been established, let’s take a look at what is happening when we link DBpedia individuals to UMBEL named entities: how that actually works to explode the domain.

Let’s take the example of dbpedia:Plato. This individual is currently defined in DBpedia as:

  • yago:AncientGreekPhysicists
  • yago:PhilosophersOfLanguage
  • yago:PhilosophersOfLaw
  • yago:PoliticalPhilosophers
  • yago:AncientGreekVegetarians
  • yago:AcademicPhilosophers
  • yago:Philosopher110423589

Fine, but what does this mean? What if my system doesn’t know any of these classes? We, as humans, know that Plato is a person, a human being. But it is totally another story for a software agent.

What we want to do here is to explode Plato’s domain to try to find a meaning that my software system can understand.

In UMBEL, the “Plato” named entity is defined as an umbel:Person and an umbel:Intellectual. If you take a look at the detailed report for these two subject concepts, you will be able to see in the section “Broader Subject Concepts” the super-classes that Plato belongs to. So we know that Plato is a social being, a homo sapiens, etc. This is basically what happens with Yago too, except that the conceptual structure (the way to describe the entity) differs.

However one thing that is happening is that we exploded Plato’s domain with classes defined in external ontologies. As you can notice in the sections “Broader External Classes” and “Equivalent External Classes”, Plato is also a: foaf:Person, a foaf:Agent and a cyc:Person.

This means that if my software agent doesn’t know what a “yago:Person100007846” means; it alternatively may know what a foaf:Person or a foaf:Agent means. And if it knows what it means, then it will be able to properly manipulate it: to display it in a special way; to refer to it as a person; so to do whatever it can with information about a “person”.

This exploding the domain works because these external ontologies classes have been referentially linked to a coherent conceptual structure.

The inference path

Let’s take a look at the fundamental reasons why the scenario above works.

First, you, and your system, have to trust the UMBEL named entities dictionaries and the UMBEL subject concept structure to perform the inference that I will explain below. If you and your system trust these linkage assertions, then you will be able to act according to the knowledge that has been inferred.

DBpedia individuals are linked to UMBEL named entities using the owl:sameAs property. This means that DBpedia individual A is identical (same semantic meaning) as the UMBEL named entity B. They both refer to the same individual.

This means that if B is defined as being of rdf:type sc:Person (“sc” stands for Subject Concept), then we can infer that A is defined as being of rdf:type sc:Person too.

If sc:Person is owl:equivalentClass with foaf:Person, we can infer that umbel:B is a foaf:Person, so that dbpedia:A is a foaf:Person too!

We can see similar examples for exploding the domains:

Exploring ConceptualWorks, PeriodicalSeries and NewspaperSeries

In my “UMBEL as a Coherent Framework to Support Ontology Development” blog post from last week, I showed how UMBEL subject concepts acted to create context for linked classes defined in external ontologies. Since DBpedia individuals are instances of classes, and that some of these classes are linked to UMBEL, these subject concept classes also give context to those individuals!

As some examples, go ahead and take a look at the “Named Entities for …” section of these detailed report pages:

The partial list of named entities that are returned by the detailed report viewer shows named entities that mainly come form Wikipedia (so that have links to DBpedia). These subject concepts gives a coherent context to those DBpedia individuals.

You should quickly notice, for example, that dbpedia:Kansas_City_Times is not only a sc:NewspaperSeries, a sc:PeriodicalSeries and a sc:ConceptualWork. You also notice that it is a frbr:Work, a bibo:Periodical and a bibo:Newspaper.

The context created by these UMBEL subject concepts gives not only new power to linked external classes, but also to their instances, such as these DBpedia individuals!

Conclusion

Contexts created by UMBEL subject concepts emerge by the power of linkage that exists between all the subject concepts, and the linkage between those subject concepts classes with classes defined in external ontologies. These contexts are consistent because of the coherence of the structure that is powered by OpenCyc (Cyc).

So far, most Linked Data has been about the “things” or named entities of the world, organized according to either Wikipedia categories or WordNet. These structures may have some internal structural consistency, but were never designed to play the role as a coherent reference framework. The coherence of UMBEL (based on the coherence of Cyc) is a powerful contextual lever for bringing order to this chaos.

Once information gets linked to a coherent framework such as UMBEL, things start to happen; powerful things. And, with each new linkage and relation to additional external ontologies, that power increases exponentially.

I wrote this blog post to show again the power of exploding the domain using DBpedia as an example, and how UMBEL can help to use and to leverage such big datasets.