Evaluation guidelines

If possible, evaluations should use the standard datasets provided and present results for the following reasoning tasks (where relevant for the system being evaluated):

• Classification. The dataset consists of a set of OWL ontologies. The total time taken to load and classify each ontology should be reported. It would also be interesting to report on comparisons of the computed taxonomy with the "reference" taxonomies that are provided with the dataset.

• Class satisfiability. The dataset consists of a set of OWL ontologies, and for each ontology one or more class URIs. The time taken to perform each test along with the satisfiability result for each class should be reported.

• Ontology satisfiability. That dataset consists of a set of OWL ontologies. The total time taken to load and test the satisfiability of each ontology should be reported, along with the satisfiability result for each ontology.

• Logical entailment. The dataset consists of a set of pairs of OWL ontologies. The total time take to determine if the first ontology entails the second ontology should be reported, along with the entailment result (true or false).

• Instance retrieval. The dataset is an OWL ontology and a class expression. For each ontology the total time taken to load the ontology and retrieve the sets of instances for each class expression should be reported. It would also be interesting to report on comparisons of the retrieved instances with the "reference" set that are provided with the dataset.

It is suggested that full results of any evaluations performed are made available via the web, with summaries of the results being included in the papers submission as space permits.

SEALS infrastructure

Optionally, participants may consider to submit their systems using the SEALS infrastructure (by April 16th). The use of SEALS platform will allow the organisers to perform standardised evaluations of the submitted systems and present the results during the workshop.

In order to do this, participants should first join the SEALS Community. Once you have your community login, you will be able to register your tools for evaluation. Please add a remark in the description of the version to indicate that you want to participate with this version in ORE 2012.