We have run the evaluation in a high performance server with 16 CPUs and allocating 15 Gb RAM. In total, 15 out of 23 participating systems/configurations have been able to cope with at least one of the tasks of the track matching problem. Optima and MEDLEY failed to complete the smallest task with a time out of 24 hours, while OMR, OntoK, ASE and WeSeE, threw an Exception during the matching process. CODI was evaluated in a different setting using only 7Gb and threw an exception related to insufficient memory when processing the smallest matching task. TOAST was not evaluated since it was only configured for the Anatomy track and it required a complex installation. LogMapLt, a string matcher that exploits the creation of an inverted file to efficiently compute correspondences, has been used as baseline.
Together with Precision, Recall, F-measure and runtimes we have also evaluated the coherence of alignments. We have reported (1) number of unsatisfiabilities when reasoning (using HermiT) with the input ontologies together with the computed mappings, (2) the ratio/degree of unsatisfiable classes with respect to the size of the merged ontology (based on the Unsatisfiability Measure proposed in [1]), and (3) an approximation of the root unsatisfiability. The root unsatisfiability aims at providing a more precise amount of errors, since many of the unsatisfiabilities may be derived (i.e., a subclass of an unsatisfiable class will also be reported as unsatisfiable). The provided approximation is based on LogMap's (incomplete) repair facility and shows the number of classes that this facility needed to repair in order to solve (most of) the unsatisfiabilities [2].
Precision, recall and F-measure have been computed with respect to the available UMLS based alignments. Systems have been ordered in terms of the average F-measure.
Note that GOMMA has also been evaluated with a configuration that exploits specialised background knowledge (GOMMA-bk). The background knowledge of GOMMA-bk involves the application of mapping composition techniques and the reuse of mappings from FMA-UMLS and NCI-UMLS. LogMap, MaasMatch and YAM++ also use different kinds of background knowledge. LogMap uses normalisations and spelling variants from the UMLS Lexicon. YAM++ and MaasMatch use the general purpose background knowledge provided by WordNet.
LogMap has also been evaluated with two configurations. LogMap's default algorithm computes an estimation of the overlapping between the input ontologies before the matching process, while LogMap-noe has this feature deactivated.
The error-free "Large BioMed 2012 silver standard" reference alignment computed by "harmonising" the output of the participating matching systems will be available soon. We will also perform a debugging of all mapping outputs using Alcomo [3] and LogMap's repair facility [2].
1. Results for the FMA-NCI matching problem
2. Results for the FMA-SNOMED matching problem
3. Results for the SNOMED-NCI matching problem
4. Summary results for the top 8 systems
5. Harmonization of the mapping outputs
[1] Christian Meilicke and Heiner Stuckenschmidt. Incoherence as a basis for measuring the quality of ontology mappings. In Proc. of 3rd International Workshop on Ontology Matching (OM), 2008. [url]
[2] Ernesto Jimenez-Ruiz and Bernardo Cuenca Grau. LogMap: Logic-based and scalable ontology matching. In Proc. of 10th International Semantic Web Conference (ISWC), 2011. [url]
[3] Christian Meilicke. Alignment Incoherence in Ontology Matching. University of Mannheim, Chair of Artificial Intelligence (2011) [url]
This track is organised by Ernesto Jimenez Ruiz, Bernardo Cuenca Grau and Ian Horrocks, and supported by the SEALS and LogMap projects. If you have any question/suggestion related to the results of this track, feel free to write an email to ernesto [at] cs [.] ox [.] ac [.] uk or ernesto [.] jimenez [.] ruiz [at] gmail [.] com