Today we released a simple structWSF nodes statistics report. It aggregates different statistics from all know (and accessible) structWSF nodes on the Web. It is still in its early stage, but aggregated statistics so far are quite interesting.
This global statistics reports has two aims:
- Monitoring the evolution of the usage of structWSF, and
- Monitoring the overall performance of structWSF web services in different setups for different usages
The report is accessible here in all time. The report is updated hourly.
Overall Statistics
The main statistics of the report are:
- The number of structWSF nodes participating to the report
- The total number of HTTP queries processed by the structWSF nodes
- The total number of datasets created on the nodes
- The total number of records indexed, and
- The total number of triples indexed
These statistics gives a general overview of the size of the “global structWSF network of nodes”.
Web Service Statistics
Each Web service endpoint has its own statistics, which are:
- The number of queries processed by the web service
- The average time it took to process the query (without the network latency between the requested and the web service endpoint server)
- All the requested mime-types, and the number of times a mime-type have been requested, and
- All the HTTP response code returned by the endpoint
These Web service specific statistics are helpful to have a general understanding of each web service endpoint.
The average time per query is helpful to know what kind of performance a developer should expect when using this web service endpoint.
The list of requested MIME types gives an overall usage of the web service endpoint: are users mostly requesting XML data, JSON data, RDF+XML data, etc. Such usage statistics is helpful to prioritize future development tasks.
The list of all HTTP response code is helpful to notice possible issues with a web service endpoint. If error codes are returned often, this could pinpoint a possible bug in the web service endpoint, an issue with its usage that could lead to a fix in the documentation, etc.
Participating to the Global structWSF Statistics Report
If you are operating a structWSF instance and want to participate to the Global structWSF Statistics Report, you first have to download the new statisticsBroker.php script and install it on your structWSF node.
The statistics broker script is what calculates the statistics of a structWSF node, and what is used to aggregate statistics from all nodes, to generate the consolidated report.
The first thing to do is to edit the file, and to change the value of the $enableStatisticsBroadcast variable from FALSE to TRUE at the line 46. This will enable the script.
Normally you should install the script in the root folder of your structWSF node, but you can install it anywhere on your server, where it will be accessible on the Web.
The final step is to register your node to the reporting system. It is just a matter of registering the URL address where the statisticsBroker.php script is accessible. It should be added to the global report within 24 hours, once I validated it.
Other Usage of the Statistics Broker
This is nice to participate to such global statistics report, but much more can be done with such a statistics broker.
A structWSF developer or a structWSF node maintainer could use it to have statistics of the local node. As described above, such statistics can be used to pinpoint possible performance issues, bottlenecks and possible bugs in web service endpoints. It could also be use to plan future extension of the network to scale some highly used web service endpoint in the network.
Additionally, the statistics broker could be used in a broader server maintenance architecture. It could be used in conjunction with another script to be part of a Ganglia monitoring system for example. Performances could be monitored by Ganglia, rate of requests per hours, raise in the number different HTTP response returned by some web services. Additionally, each of these statistics could be bound to different alerts notification messages that would alert the structWSF system maintainers and developers of possible issues with the network.
Next Step
The next step with the statistics broker will be to create a structWSF web service out of it. That way, structWSF node maintainers will be easily able to define access and usage permissions for such statistics.