Ontology Alignment Evaluation Initiative - OAEI-2017 Campaign

Large BioMed Track

OAEI 2017::Large BioMed Track

General description

This track consists of finding alignments between the Foundational Model of Anatomy (FMA), SNOMED CT, and the National Cancer Institute Thesaurus (NCI). These ontologies are semantically rich and contain tens of thousands of classes.

UMLS Metathesaurus has been selected as the basis for the track reference alignments. UMLS is currently the most comprehensive effort for integrating independently-developed medical thesauri and ontologies, including FMA, SNOMED CT, and NCI.


Modalities and SEALS support

This track has two main objectives. On the one hand, it intends to evaluate the performance of matching systems when matching real large scale ontologies (Modality 1). On the other hand, it also aims at creating an error free reference alignment; to this end, mapping repair systems are also welcome to participate in the track (Modality 2).

Regarding the use of background knowledge, the OAEI rules state that a resource (i.e. a third biomedical ontology) especially designed for the test is not allowed. Particularly, matching systems using the UMLS Metathesaurus as background knowledge will have a notable advantage since the reference alignment is also based on the UMLS Metathesaurus.

Modality 1: standard matching

For this modality the generated alignment should be an optimal solution to the matching problem with respect to both recall and precision with respect to the 2017 reference alignment.

As in 2014 and 2015, instead of repairing the original UMLS reference alignments by removing mappings leading to unsatisfiable classes, we flagged the incoherence-causing mappings in the alignment by setting the relation to "?" (unknown). These "?" mappings will neither be considered as positive nor as negative when evaluating the participating ontology matching systems, but will simply be ignored. This way, systems that do not perform mapping repair are not penalized for finding mappings that (despite causing incoherences) may or may not be correct, and systems that do perform mapping repair are not penalized for removing such mappings either.

Nevertheless, we will still give a special attention to the number of unsatisfiabilities caused by the mappings computed by a participating system. Thus, we encourage system developers to implement mapping repair techniques or reuse state-of-the art techniques.

The evaluation of Modality 1 will be run with support of the SEALS infrastructure. This requires that you wrap your matching system in a way that allows us to execute it using the SEALS client (see OAEI 2017 evaluation details).

The complete datasets for the OAEI 2017 campaign can be downloaded as a zip file (LargeBioMed_dataset_oaei2017.zip [17Mb]) or accessed via the SEALS platform (see dataset identifiers below).

Modality 2: mapping repair (optional)

Mapping repair systems are also welcome to provide a revised version of the original UMLS mappings to contribute in the creation of future reference alignments. Currently, Alcomo, LogMap and AML have contributed in the 2017 reference alignment.

Modality 2 will be optional and will be run in an 'off-line' way.


Data sets

The Large BioMed Track consists of several matching tasks involving different fragments of the FMA, NCI and SNOMED CT ontologies.

Note that the ontologies have been normalised for the OAEI, as a result the synonyms of concept names are provided as "rdfs:label" annotations.

Required input for the SEALS OMT client:


All Largebio matching tasks

With the following Version-ID all 6 largebio matching tasks are executed. For individual matching tasks refer to the ids below.


FMA-NCI matching problem

Task 1: FMA-NCI small fragments

This task consists of matching two (relatively) small fragments of FMA and NCI. The FMA fragment contains 3,696 classes (5% of FMA), while the NCI fragment contains 6,488 classes (10% of NCI).

Task 2: FMA-NCI whole ontologies

This task consists of matching the whole FMA and NCI ontologies, which contains 78,989 and 66,724 classes, respectively.


FMA-SNOMED matching problem

Task 3: FMA-SNOMED small fragments

This task consists of matching two (relatively) small fragments of FMA and SNOMED. The FMA fragment contains 10,157 classes (13% of FMA), while the SNOMED fragment contains 13,412 classes (5% of SNOMED).

Task 4: FMA whole ontology with SNOMED large fragment

This task consists of matching the whole FMA that contains 78,989 classes with a large SNOMED fragment that contains 122,464 classes (40% of SNOMED).


SNOMED-NCI matching problem

Task 5: SNOMED-NCI small fragments

This task consists of matching two (relatively) small fragments of SNOMED and NCI. The SNOMED fragment contains 51,128 classes (17% of SNOMED), while the NCI fragment contains 23,958 classes (36% of NCI).

Task 6: NCI whole ontology with SNOMED large fragment

This task consists of matching the whole NCI that contains 66,724 classes with a large SNOMED fragment that contains 122,464 classes (40% of SNOMED).