Ontology Alignment Evaluation Initiative - OAEI-2017 Campaign

Large BioMed Track

Results OAEI 2017::Large BioMed Track

Contact

If you have any question/suggestion related to the results of this track or if you notice any kind of error (wrong numbers, incorrect information on a matching system, etc.), feel free to write an email to ernesto [at] cs [.] ox [.] ac [.] uk or ernesto [.] jimenez [.] ruiz [at] gmail [.] com

Evaluation setting

We have run the evaluation in a Ubuntu Laptop with an Intel Core i7-4600U CPU @ 2.10GHz x 4 and allocating 15Gb of RAM.

Precision, Recall and F-measure have been computed with respect to a UMLS-based reference alignment. Systems have been ordered in terms of F-measure.

Participation and success

In the OAEI 2017 largebio track 10 out of 21 participating systems have been able to cope with at least one of the tasks of the largebio track with a 4 hours timeout. Note that we also include the results of Tool1 (the developers withdrew the system from the campaign) as reference.

Use of background knowledge

LogMapBio uses BioPortal as mediating ontology provider, that is, it retrieves from BioPortal the most suitable top-10 ontologies for the matching task.

LogMap uses normalisations and spelling variants from the general (biomedical) purpose UMLS Lexicon.

AML has three sources of background knowledge which can be used as mediators between the input ontologies: the Uber Anatomy Ontology (Uberon), the Human Disease Ontology (DOID) and the Medical Subject Headings (MeSH).

YAM-BIO uses as background knowledge a file containing mappings from the DOID and UBERON ontologies to other ontologies like FMA, NCI or SNOMED CT.

XMAP uses synonyms provided by the UMLS Metathesaurus. Note that matching systems using UMLS-Metathesaurus as background knowledge will have a notable advantage since the largebio reference alignment is also based on the UMLS-Metathesaurus.

Alignment coherence

Together with Precision, Recall, F-measure and Runtimes we have also evaluated the coherence of alignments. We have reported (1) number of unsatisfiabilities when reasoning with the input ontologies together with the computed mappings, and (2) the ratio/degree of unsatisfiable classes with respect to the size of the union of the input ontologies.

We have used the OWL 2 reasoner HermiT to compute the number of unsatisfiable classes. For the cases in which HermiT could not cope with the input ontologies and the mappings (in less than 2 hours) we have provided a lower bound on the number of unsatisfiable classes (indicated by ≥) using the OWL 2 EL reasoner ELK.

In this OAEI edition, only three systems have shown mapping repair facilities, namely: AML, LogMap (including LogMapBio variant) and XMap. AML and XMap, as LogMap, produce relatively clean outputs in FMA-NCI and FMA-SNOMED cases; however in the SNOMED-NCI cases their mappings lead to a number of unsatisfiable classes. The results also show that even the most precise alignment sets may lead to a huge amount of unsatisfiable classes. This proves the importance of using techniques to assess the coherence of the generated alignments.


1. System runtimes and task completion

System FMA-NCI FMA-SNOMED SNOMED-NCI Average # Tasks
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6
LogMapLite 1 10 2 18 9 22 10 6
AML 44 77 109 177 669 312 231 6
LogMap 12 92 57 477 207 652 250 6
XMap* 20 130 62 625 106 563 251 6
YAM-BIO 56 279 60 468 2,202 490 593 6
Tool1 65 1,650 245 2,140 481 1,150 955 6
LogMapBio 1,098 1,552 1,223 2,951 2,779 4,728 2,389 6
POMAP 595 - 1,841 - - - 1,218 2
SANOM 679 - 3,123 - - - 1,901 2
KEPLER 601 - 3,378 - - - 1,990 2
WikiV3 108,953 - - - - - 108,953 1
# Systems 11 7 10 7 7 7 10,795 49
Table 1: System runtimes (s) and task completion.


2. Results for the FMA-NCI matching problem

Task 1: FMA-NCI small fragments

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 20 2,649 0.977 0.901 0.937 2 0.019%
YAM-BIO 56 2,681 0.969 0.896 0.931 800 7.8%
AML 44 2,723 0.958 0.910 0.930 2 0.019%
LogMapBio 1,098 2,807 0.934 0.906 0.920 2 0.019%
LogMap 12 2,747 0.944 0.897 0.920 2 0.019%
KEPLER 601 2,506 0.960 0.831 0.891 3,707 36.1%
Average 10,193 2,550 0.946 0.844 0.891 1,238 12.0%
LogMapLite 1 2,483 0.967 0.819 0.887 2,045 19.9%
SANOM 679 2,457 0.947 0.803 0.869 1,183 11.5%
POMAP 595 2,475 0.898 0.827 0.861 3,493 34.0%
Tool1 65 2,316 0.974 0.767 0.858 1,128 11.0%
WikiV3 108,953 2,210 0.883 0.726 0.797 1,261 12.3%
Table 2: Results for the largebio task 1.

Task 2: FMA-NCI whole ontologies

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 130 2,735 0.884 0.847 0.865 9 0.006%
AML 77 2,968 0.838 0.872 0.855 10 0.007%
YAM-BIO 279 3,109 0.818 0.888 0.852 11,770 8.1%
LogMap 92 2,701 0.856 0.808 0.831 9 0.006%
LogMapBio 1,552 2,913 0.817 0.834 0.825 9 0.006%
Average 541 2,994 0.797 0.830 0.812 7,389 5.1%
LogMapLite 10 3,477 0.673 0.820 0.739 26,478 18.1%
Tool1 1,650 3,056 0.692 0.738 0.714 13,442 9.2%
Table 3: Results for the largebio task 2.


3. Results for the FMA-SNOMED matching problem

Task 3: FMA-SNOMED small fragments

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 62 7,400 0.974 0.847 0.906 0 0.000%
AML 109 6,988 0.923 0.762 0.835 0 0.000%
YAM-BIO 60 6,817 0.966 0.733 0.834 13,240 56.1%
LogMapBio 1,223 6,315 0.946 0.693 0.800 1 0.004%
LogMap 57 6,282 0.947 0.690 0.798 1 0.004%
Average 1,010 4,623 0.891 0.510 0.615 2,141 9.1%
KEPLER 3,378 4,005 0.822 0.424 0.559 3,335 14.1%
SANOM 3,123 3,146 0.688 0.304 0.422 2,768 11.7%
POMAP 1,841 2,655 0.682 0.299 0.416 1,013 4.3%
LogMapLite 2 1,644 0.968 0.209 0.344 771 3.3%
Tool1 245 979 0.990 0.135 0.238 287 1.2%
Table 4: Results for the largebio task 3.

Task 4: FMA whole ontology with SNOMED large fragment

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
XMap* 625 8,665 0.774 0.843 0.807 0 0.000%
YAM-BIO 468 7,171 0.887 0.728 0.800 54,081 26.8%
AML 177 6,571 0.882 0.687 0.772 0 0.000%
LogMap 477 6,394 0.840 0.645 0.730 0 0.000%
LogMapBio 2,951 6,634 0.806 0.650 0.720 0 0.000%
Average 979 5,470 0.845 0.556 0.627 8,445 4.2%
LogMapLite 18 1,822 0.852 0.209 0.336 4,389 2.2%
Tool1 2,140 1,038 0.873 0.129 0.225 649 0.3%
Table 5: Results for the largebio task 4.


4. Results for the SNOMED-NCI matching problem

Task 5: SNOMED-NCI small fragments

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 669 14,740 0.871 0.746 0.804 ≥3,966 ≥5.3%
LogMap 207 12,414 0.947 0.690 0.798 ≥0 ≥0.000%
LogMapBio 2,779 13,205 0.889 0.682 0.772 ≥0 ≥0.000%
YAM-BIO 2,202 12,959 0.899 0.677 0.772 ≥549 ≥0.7%
Average 921 12,220 0.895 0.592 0.697 21,264 28.3%
XMap* 106 16,968 0.894 0.566 0.693 ≥46,091 ≥61.3%
LogMapLite 9 10,942 0.892 0.567 0.693 ≥60,450 ≥80.4%
Tool1 481 4,312 0.871 0.218 0.349 ≥37,797 ≥50.2%
Table 6: Results for the largebio task 5.


Task 6: NCI whole ontology with SNOMED large fragment

System Time (s) # Mappings Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 312 13,176 0.904 0.668 0.768 ≥720 ≥0.4%
YAM-BIO 490 15,027 0.827 0.698 0.757 ≥2,212 ≥1.2%
LogMapBio 4,728 13,677 0.835 0.640 0.725 ≥5 ≥0.003%
LogMap 652 12,273 0.868 0.597 0.707 ≥3 ≥0.002%
LogMapLite 22 12,894 0.797 0.567 0.663 ≥150,656 ≥79.5%
Average 1,131 13,666 0.837 0.563 0.660 55,496 29.3%
XMap* 563 23,707 0.819 0.553 0.660 ≥137,136 ≥72.4%
Tool1 1,150 4,911 0.812 0.215 0.340 ≥97,743 ≥51.6%
Table 7: Results for the largebio task 6.

* Uses background knowledge based on the UMLS-Metathesaurus as the largebio reference alignments.