Eight months ago we announced the dissolution of Zitgist LLC. This event led to the creation of a “sandbox“ to keep alive all the online assets of the company. Since this sandbox server was not owned by Structured Dynamics, it was becoming hard for us to update UMBEL and its online services. It is why we took the time to move the services back on to our new servers.

A New Home

Structured Dynamics LLC now hosts a new version for the UMBEL Web services. From the main menu at the SD Web site you can access these services under the “umbel ws” menu option (you can also bookmark the Web services site at umbel.structureddynamics.com or ws.umbel.org.)

This move of UMBEL’s Web services to a new home will make the future upgrade of UMBEL easier, and this will make the maintenance of the Web services endpoints easier as well. With this move, I am pleased to announce the release of five initial Web services and one visualization tool:

Lookup Web Services:

• Finder: Subject Concept
• Reporter: Subject Concept

Inference Engine Web Services:

• Inference: Lister — list sub-classes, super-classes and equivalent-classes
• Inference: Validator — verify sub-class, super-class and equivalent-class relationships

SPARQL endpoint Web Service:

• SPARQL Endpoint

Visual Tool:

• Subject Concept Explorer

Note that the visual tool is using Moritz Stefaner’s Relation Browser.

Ping the Semantic Web

Additionally, the Ping the Semantic Web RDF pinging service is now the property of OpenLink Software Inc. OpenLink is now hosting, maintaining and developing the service.

Pinging and receiving list of newly created and updated RDF resources has never been easier! I am pleased to announce the release of the latest version of Ping the Semantic Web.

In this brand new system you have access to a:

1. Validated RDF resources
2. Simplified pings list export system
3. Faster pinging infrastructure
4. Brand new user interface
5. New statistics

1. Validated RDF resources

In the version 2.0, PTSW was doing a pseudo validation of RDF files. In the version 3.0, it fully validates RDF documents. This means that all pings the service export are valid RDF documents.

This is a major upgrade to the system since now all agents requesting pings from PTSW will know that each of them are valid RDF documents. That way, they will save time and bandwidth since they won’t try to process bad RDF documents.

2. Simplified pings list export system

Now all ping consumers need to be registered to the PTSW web service. This simple registration greatly helps consuming pings coming from PTSW. There are the steps to follow to get pings from the server:

1. The user have to register an account on pingthesemanticweb.com
2. He has to register the IP address of the server that will download the xml file listing all the latest pings received from the system
3. Additionally, he has to setup his pings retrieval preferences in the user account section.
4. The registered web server has to request the xml file at: http://pingthesemanticweb.com/export/

What improved is the way applications get pings. Now a web server only has to request the xml file, and PTSW will take care to created the xml file according to the user’s preferences.

Finally, PTSW is archiving the time of the latest request of the user. Next time the user’s server will request this document, it will receive the xml file with all the pings received by PTSW since its last request.

This is a major improvement since if the user’s web server was down for 2 days, for some reason, it won’t lose any pings since PTSW will send him all the pings received by the service in the last 2 days.

Note: all current Ping the Semantic Web ping consumers have to create an account and change their application accordingly.

3. Faster pinging infrastructure

The web service is now hosted on a much bigger server. We also switched from MySQL to Virtuoso. These changes result in a more powerful service that I estimate to be able to handle up to 5 million pings per day (in the best of the World with fast remote web server delivering the RDF content). In any case, it is probably enough for the next year’s expansion.

4. Brand new user interface

We also spent some time refreshing the user interface of the web service. This new interface will help us to easily integrate new features and sections to the service’s web site along with keeping it appealing to users.

5. New statistics

New statistics on the state of the service are now available.

1. All stats about Namespaces. This is the list of namespaces used to describe entities in RDF. For example, if a RDF document has an entity types as a sioc:Post, then the SIOC namespace will be added and its stat counter will be incremented by one. There is currently 347 used namespaces know by PTSW.
2. All stats about Types. This is the number of typed entities defined in each RDF document know by PTSW. For example, if a RDF document has four foaf:Person defined, then four will be added to the counter. If the same entity (URI) is defined in two different RDF documents, the type of the entity will be calculated twice. So take these numbers as a good approximation, but not as an absolute truth. There is currently 2773 types know by PTSW.

Some people will notice that the current numbers in the sidebar are completely different from the ones that were on the old website of the service. They are right, and there is the reason: I pruned the geonames.org and talkdigger.com pings from PTSW.

In fact, when I started the web service, I added these two RDF data dumps to PTSW. At that time, initiatives such as the Linking Open Data Community didn’t exist and people didn’t know how to export their RDF data dumps. So I choose to include them in the PTSW system. Since then, methods improved and things changed. Now RDF data dumps are available directly from these web sites, data dump repository exists, and people don’t use PTSW for that reason. In fact, they use the PTSW exportation feature to synch their service, and not to get complete datasets from them. This said, I pruned all these 7 000 000 documents from the system leaving about 845 000 “wild” RDF documents in the system.

It is the inclusion of these complete data sources that were increasing the stats compared to today’s stats.

Conclusion

When I created Ping the Semantic Web more than one year ago, I hoped developers would use the service to easily find RDF data without crawling the entire. I hoped that this web service would be a vector of semantic web application development. I think that it succeeded in some ways when I think about services such as SIndice and DOAPStore that emerged from the PTSW initiative.

This new version of Ping the Semantic Web tries to go further in that directing: making thinks even simpler for RDF data consumer and giving them a more powerful RDF discovery service.

Note: make sure to refresh the DNS cache of your desktops and servers so that you see the new, and not the old, PTSW web site.

Three weeks ago, Rob Cakebread contacted me vis-à-vis some possible issues with the PingtheSemanticWeb.com web service. Rob is involved with the development of Gentoo Linux and he wanted to come-up with a method to let their development teams know when new packages are released for some projects. Since the RSS feeds of Freshmeat.net are limited in length, and that they do not support DOAP, he started to think about a way to convert Freshmeat.net’s data projects into RDF using the DOAP ontology. That way, he could easily create services to track the release of new packages and then increasing the effectiveness of their development teams.

Rob wrote:

I came across ptsw.com when trying to determine how many package indexes are using DOAP right now (so far I’ve only found Python Package Index and O’Reilly’s Code Zoo).

I’m a developer for Gentoo Linux. A website I created (Meatoo) checks Freshmeat twice a day and finds packages that have new releases available. We have a database of all our maintainers grouped into ‘herds’, a Python herd, desktop herd, PHP herd etc. The developers in the herds can query my website by a command-line client that uses XML-RPC, or subscribe to RSS feeds by herd or package name, or read the website itself and see which packages have new releases.

DOAP fits into this because I was thinking about creating DOAP records for each release from each package index and making this available so people can write tools to find out information about software packages easily.

Its how we got in contact. Rob had a practical problem, then he tried to find a way to resolve it and to help other people to resolve it too; and its how he found PingtheSemanticWeb.com and other semantic web related projects and communities (such as the Linked Open Data).

Freshmeat.net in DOAP

Then a couple of days ago, Rob re-contacted me to let me know that the Freshmeat.net’s 43 000 projects description is now available in RDF.

He created a prototype service that converts the data he has aggregated from Freshmeat.net into DOAP. Its project is called DOAPSpace. The idea is to make available the Freshmeat.net projects into DOAP, then to ping PingtheSemanticWeb.com to ultimately make them available on Doapstore.org that is feeded by PTSW.

People can get the DOAP description of each project by going to:

http://doapspace.gentooexperimental.org/doap/<project_name>

There are some examples of URIs:

http://doapspace.gentooexperimental.org/doap/0verkill
http://doapspace.gentooexperimental.org/doap/amanda

RDF dump

I was really pleased to see how Rob managed to generate that data. Then I asked him if a RDF dump of that precious data would eventually be available for download? It is exactly what he is doing at the moment, and as soon as he send me the dump, I will make it available via PingtheSemanticWeb.com. Then, it will be ready to be integrated into the Linking Open Data project.

Content Negotiation

At the same time, Rob added a new feature to its service; a user only has to append the “?zg=1″ parameter to the URL to get redirected to the Zitgist Browser. It was really nice from him to think about that; I really appreciated.

However, I introduced him at how he could use content-negotiation to do that and to make its service compatible with other tools such as other RDF browsers. So I pointed him to the How to Publish Linked Data on the Web draft document so that he can have a better understanding of the content-negotiation process.

Linking Open Data Community and Early Developers

Rob is certainly an early adopter of the Semantic Web. He is a developer that wants to solve problems with methods and technologies. He had the intuition that the DOAP ontology and other semantic web principles and technologies could help him to solve one of its problems. This intuition leaded him to discover what the semantic web community could do to help him.

It’s the kind of user we have to take care of; and that we have to help to release their projects. Its people like Rob that will make the Semantic Web a reality. Without such early adopters, from outside of the Semantic Web Community, the semantic web is probably doomed. We are there now; ready to help developers to integrate semantic web technologies into their projects; to generate their data into RDF and to link it with other data sources. It’s the goal of communities like the Linking Open Data Community and its what we are about to do.

It is not because we had some issues with the Zitgist Browser‘s server that things stopped at Zitgist. In fact, many projects evolved at the same time and I outline some of these evolutions bellow.

New version of the Zitgist Browser

A new version of the browser is already on the way. In fact, the pre-release version of the browser was a use case; a prototype. Now that we know that it works and that we faced most of the issues that have to be taken into account to develop such a service, we hired Christopher Stewart to work on the next version of the browser. He is already well into the problem now, so you could expect a release of this new version sooner than you could be expecting. At first, there won’t be many modifications at the user interface level, however, many things will be introduced in this new version that will help us to push the service at another level in the future.

New version of Ping the Semantic Web

The version 3.0 of the PingtheSemanticWeb.com web service should be put online next week. It will be a totally new version of the service. It won’t use MySQL anymore; Virtuoso has replaced it. The service will now fully validate RDF files before including them in the index. More stats will be available too. It is much faster (as long as remote servers are fast too) and I estimate that this only server could handle between 5 to 10 million pings per day (enough for the next year’s expansion). This said, the service will be pushed at another level and be ready for more serious traffic. After its release, a daily dump of all links will be produced as well.

The first draft of the Bibliographic Ontology

The Bibliographic Ontology Specification Group is on fire. We are now 55 members and generated 264 posts in July only. Many things are going on here and the ontology is well underway. We should expect to release a first draft of the ontology sometime in August. If you are interested in bibliographic things, I think it’s a good place to be.

The Zitgist Semantic Web Query Service

Finally, Zitgist’s Semantic Web Query Service should be available for alpha subscribed users sometime in September. You can register to get your account here. Also, take a look at what I wrote about vis-à-vis this search module (many things evolved since, but it’s a good introduction to the service).

Conclusion

So, many things are going on at Zitgist and many exiting things should happen this autumn, so stay tuned!

Zotero is a great FireFox add-on that lets its users find, search, edit and create citations they find on the Web while browsing it. All the power of Zotero resides in its “translation modules“. These modules will detect citations in various types of web pages. When it detects one of these citations, it will notify its users to give them the opportunity to save them.

What interest me is that Zotero already use some ontologies to export users’ citations libraries using RDF. When I noticed that I started to wonder: what could we do with Zotero now?

The Zotero vision

Zotero is the best-integrated citation tool for the Web I know. A phenomenal amount of citations can be discovered on the Web via Zotero users community.

Remember what we have done with the Semantic Radar a couple of months ago? This FireFox add-on was detecting SIOC RDF documents in Web page. Then I contacted Uldis Bojar to ask him to ping PingtheSemanticWeb.com each time a user was detecting a RDF file while he was browsing the Web. Now a good source of RDF data pinged to PTSW come from Semantic Radar users. This is a sort of “social semantic web discovering” technique.

What I would like to do is the same thing but for Zotero.

[Click to enlarge to full size]

1. Zotero users browse the Web, discover citations and save them into their personal libraries.
2. Each time a Zotero instance discover a citation, it would send the URL where we can find it to PingtheSemanticWeb.com.
   1. Note: the user should be aware of that functionality via an option into Zotero that would explains him what this feature it is all about, and to gives him the possibility to disable it.
   2. Note: Zotero would ping PTSW each time it detects a citation (so that the icon appears in the FireFox’s URL bar), and not each time a user save it.
3. Via the Virtuoso Sponger, PingtheSemanticWeb.com will check the incoming URL from Zotero users and will check to find citations too. If a citation is found, it will be added to its list of know citations and archive their content.
4. PingtheSemanticWeb.com will then send the new citations to Zitgist so that it can include them into its database.
   1. Note: here Zitgist could be replaced by any web service wanting them. Remember that PTSW act as a data-multiplexer.
5. Via Zitgist (that is a semantic web search engine), users from around the World will be able to search among these citations (discovered by Zotero users) and to browse them.

Zitgist has a Zotero citation provider

What is fantastic here is that Zitgist become a source of citations. So if a Zitgist user has Zotero installed, then he will be able to batch-save the list of results returned by Zitgist; and if the user is browsing Zitgist’s citations, he will be able to include them into their Zotero instance like if Zitgist would be Amazon.com or any other citations web sites.

That way, Zotero’s found data would be accessible to Zotero users via Zitgist that would then become a citations provider (mainly feed by the Zotero community).

You see the interaction?

What have to be developed?

Some things have to be developed to make that vision working. No major development, but only a couple of features to develop on each system.

Integration of Ping the Semantic Web into Zotero

The integration of Ping the Semantic Web into Zotero is quite straightforward.

Pinging PingtheSemanticWeb.com via a web service

The first step is to make Zotero notify PTSW each time it comes across a citation. It has to send the URL of that/these citation(s) via XML-RPC or REST.

That is it. Each time Zotero detect a citation, it sends a simple ping to PTSW via an XML-RPC or REST request.

Adding a pinging option to Zotero

Another thing that Zotero would have to add to their add-on is an option that would gives the possibility to their users to disable that feature in case they don’t want to send a notification to PTSW each time they discover a citation on a Web page while they are browsing the Web.

Development of Zotero translators into Sponger Metadata Cartridge

The biggest development effort that would have to be done is to convert the Zotero translators into Virtuoso Sponger’s Metadata Cartridge.

Right now, Metadata Cartridge exists for: Google Base, Flickr, microformats (hReview, hCalendar, etc.), etc. These cartridges are the same things as “Zotero translators” but for the Virtuoso Sponger. By developing these cartridges, everybody running Virtuoso will be able to see these citations (from Amazon, etc.) as RDF data (mapped using some ontologies).

Documentation about how to develop these cartridges will be available in the coming days. From there, we would be able to setup an effort to convert the Zotero Translators into Spongers Metadata Cartridges.

Conclusion

This is the vision I have of the integration of Zotero into the current Semantic Web environment that exists. Any ideas, suggestions, collaboration propositions would be warmly welcome.

Note: a discussion about this subject started on Zotero’s web forum 

Three days ago I talked about the importance of both RDF data dumps and dereferencable URIs to distribute RDF data over the Web. However yesterday Marc from Geonames.org got some problems with an impolite semantic web crawler. In his article he point out that:

“It simply does not make sense to download a huge database record by record if a full dump is available.”

In the best of the world it doesn’t make sense, but unfortunately it is how the Web always worked. Think about Google, Yahoo! and MSN Search; this is exactly what they do, and it doesn’t make sense. The difference is that they are probably more polite. Marc did the only thing he as to do: banning the belligerent crawler.

The problem with data dumps is that they are generally not that easy to find (if available) on services web site. So some developers will note care taking the time to find them and will fetch everything from the Web server, page by page.

However all that story brings a question: how could we make these data dump more visible? Ecademy.com use a <link> element from their home page to link to a dump of the URLs to their FOAF profiles. However, if you don’t check at the HTML code of the page, you will never be aware of it. A first step would probably be to create a repository of these data dumps.

The SWEO Community Project started the “Linking Open Data on the Semantic Web” project that is basically a list of RDF dumps from different web site or projects.

Personally what I will do to help people finding these RDF dumps (and to make them aware of their existence) is to create a repository of these RDF dump on Pingthesemanticweb.com (should be available later this week).

That way, developers using Pingthesemanticweb.com will probably check that list and then download the data they need. After that, they will only use PTSW to synch their triple store with the remote service’s database (Geonames.org for example).

Technorati: Semantic | web | data | dump | rdf | crawler | spam |

In a recent mail thread, someone was asking what was the best way to get RDF data from a source [having more than a couple of thousands of documents]: a RDF dump or a list of dereferencable URIs?

None is better than the other. Personally what I prefer is to use both.

If we take the example of Geonames.org, getting all the 6.4 million of RDF documents from dereferencable URI would take weeks. However, updating your triple store with new or updated RDF documents with a RDF dump would force you to download and re-index it completely every month (or so). This task would take some days.

So what is best way then? There is what I propose (and currently do):

In fact, the first time I indexed Geonames into a triple store, I requested a RDF dump to Marc. Then I asked him: would it be possible for you to ping Pingthesemanticweb.com each time a new document appears on Geonames or each time a document is updated? In less than a couple of hour he answered to my mail and then Geonames was pinging PTSW.

So, what it means? It means that I populated my triple store with geonames with a RDF dump for the first time. By proceeding that way I saved one to two weeks of work. Then I am now updating the triple store via Pingthesemanticweb.com. By proceeding that way, I save 2 or 3 days each month.

So what I suggest is to use both methods. The important point here is that Pingthesemanticweb.com acts as an agent that send you new and updated files for a specific service (Geonames in the above example). This simple infrastructure could save precious time to many semantic web developers.

Technorati: Uri | rdf | dump | geonames | pingthesemanticweb | semantic | web |

A couple of days ago, the first web service(that I am aware of) developed using Ping the Semantic Web data has been made public.

Doap:store is a DOAP (Description Of A Project) documents search, browsing and visualization tool. It has been developed by Alexandre Passant, one of the most active contributors of the SIOC ontology.

From Alex's Blog, Doap:Store is:

Then, doap:store provides a common search engine and browsing interface for these decentralized project description, while authors keep control over their data. Data is updated each time PTSW has a new ping for it (in the future, PTSW should store new pings only if the document has changed, so updated will be made only for real document updates).

That is it: a web service helping people to find projects depending on some criteria. At the time I wrote this article, the web service focus mainly on computer projects involving programming languages. If you check at the bottom of the page, you will see categories of projects related with the programming language(s) involved in the project.

These categories are dynamically generated depending on the DOAP documents it aggregates. So as soon as new document from Ping the Semantic Web are aggregated by Doap:store, they will have an impact on this feature of the service depending on the languages defined for these projects description.

Doap:store and Ping the Semantic Web

This new web service is the perfect example of the utility of Ping the Semantic Web. With data in hands, a developer can create wonderful systems and user interface to manipulate, manage and search that data.

In this example, Alex didn't had to care about where to find DOAP documents, he only had to retrieve the list of latest created/updated DOAP documents from Ping the Semantic Web service (he do that each hour).

Keeping control over its data

This is one of the more important observations of Alex. Such an information infrastructure let the user doing what he wants with the data he generates.

This is the idea I had in mind when I wrote a blog post called "Communities’ websites should use FOAF profiles to help users managing their online persona" and this is the reason why I create an import/export feature for users's profiles using FOAF document: to give back the power to users.

Conclusion

Doap:store really shows how Ping the Semantic Web can be use by any developer. I hope other people will start to use the service the same way Alex does. Ultimately I hope that Ping the Semantic Web will become a vector of development for semantic web projects.

Technorati: Pingthesemanticweb | doapstore | doap | semantic | web | web3.0 | rdf | foaf | |

The “pings exportation” feature of Ping the Semantic Web was a little bit messy and I was really not satisfied with it. So I took the time to re-work it and I think I came up with something much better (probably something that people were expecting from the beginning).

The new way to request pings

The new way to request a list of pings from Ping the Semantic Web is quite simple. You have a set of pings (all pings received by the service so far) and you apply constraints on that set to get the subset of pings you really want for your application.

There are 7 different constraints you can apply:

1. Constraint pings for a specific type of RDF document: SIOC, FOAF, DOAP, RDFS or OWL
2. Constraint pings for a specific serialization language: XML or N3
3. Constraint pings for a time frame: last hour, yesterday or any time
4. Constraint pings with a number of results: 0 to x
5. Constraint pings for a specific domain name, example: getting all the pings from www.talkdigger.com
6. Constraint pings for a specific namespace, example: getting all the pings where the namespace “http://purl.org/dc/elements/1.1/

This new method is much more powerful. That way you can easily get a specific subset of pings for the specialized needs of your web services or software agent.

The new way to handle namespaces

Reworking this feature leaded me to rework the way Ping the Semantic Web was handling namespaces.

Now all the namespaces of the RDF documents aggregated by the service are aggregated by the service as well.

This means two things:

1. You can get RDF documents defining a specialized namespace
2. You can take a look at the list of namespaces know by Ping the Semantic Web

For the moment the service know about 400 namespaces, but it is discovering them at a rapid pace.

Conclusion

I am stabilizing the system right now and the redevelopment of this feature was resulting from that stabilization. All my updates are mostly finished and soon enough a first version of a SPARQL endpoint (and user interface) should be publicly available.

Technorati: Pingthesemanticweb | ping | semantic | web | rdf | export | service | web3.0 |

More and more web services start to export some of their data archived in their databases using RDF. Some of them have a specific goal in mind, others only do it "in case of" that someone would need it (like livejournal.com does). In any case, these RDF documents are waiting here, somewhere on the Web, waiting to be read and used.

Ping the Semantic Web's goal is to act as a central point in that environment: aggregating these RDF documents and then sending them to other services (softwares) that need them.

More and more people are starting to ping the service. It now gets about 5000 ping requests each day and its constantly growing.

In the last couple of days, I contacted some people that are developing systems that export RDF data. I asked them two things:

1. Would it be possible for you to make your system pinging Ping the Semantic Web each time it creates or updates a RDF document?
2. Would it is possible for you to send me a list of URLs where Ping the Semantic Web could find the existing RDF documents generated by your service?

Today I am asking your help:

If you know a web service that export RDF documents, would it be possible for you to contact me with the name of this service?

Then what I'll do is contacting them to ask them the two questions. What I'll also do is helping them (technically) to implement the feature into their system if they encounter any problem.

Currently the service know about 57 000 RDF documents. I predict you that in one year it will knows millions of RDF documents. And what I hope is that many of them will not be serialized using XML but using N3 instead (but I should certainly wait after a wider adoption of SPARQL before seeing that happening).

People I already contacted:

• D2R Server publishing the DBLP Bibliography Database (Richard sent me 1.2 million of URL to crawl and he should start to ping PTSW in January (yeah, this is partly why I said that I would had millions of RDF document )
• Revyu (Tom should start to ping PTSW soon)
• FOAFMap
• FOAFNaut
• Geonames (Just when I was about to publish this article, Marc contacted me telling me that Geonames now pings PTSW each time a geoname changes or is created. Also, he send me a list of 6.2 million of geonames to include).
• Tribe
• Semantic Media Wiki

Conclusion

Finally what I am requesting is help from people to try to find as many web services that export data using RDF as possible. That way everybody will benefit from it: they will increase the visibility of the data they are generating and PTSW will see its database of RDF document growing and growing for the benefit of the community.

Technorati: Pingthesemanticweb | semantic | web | web3.0 | rdf | xml | n3 | foaf | sioc | doap | rdfs | owl |