Ontology Alignment Evaluation Initiative - OAEI-2018 Campaign

Large BioMed Track

Results OAEI 2018::Large BioMed Track

Contact

If you have any question/suggestion related to the results of this track or if you notice any kind of error (wrong numbers, incorrect information on a matching system, etc.), feel free to write an email to ernesto [.] jimenez [.] ruiz [at] gmail [.] com

Evaluation setting

We have run the evaluation in a Ubuntu 18 Laptop with an Intel Core i9-8950HK CPU @ 2.90GHz x 12 and allocating 25Gb of RAM.

Precision, Recall and F-measure have been computed with respect to a UMLS-based reference alignment. Systems have been ordered in terms of F-measure.

Unique mappings show the number of mappings computed by a system that are not predicted by any of the other participants (including variants).

Check out the supporting scripts to reproduce the evaluation: https://github.com/ernestojimenezruiz/oaei-evaluation

Participation and success

In the OAEI 2018 largebio track 10 out of 18 participating systems have been able to complete at least one of the tasks of the largebio track within a 6 hours timeout. Seven systems were able to complete all six largebio tracks.

Use of background knowledge

LogMapBio uses BioPortal as mediating ontology provider, that is, it retrieves from BioPortal the most suitable top-10 ontologies for the matching task.

LogMap uses normalisations and spelling variants from the general (biomedical) purpose SPECIALIST Lexicon.

AML has three sources of background knowledge which can be used as mediators between the input ontologies: the Uber Anatomy Ontology (Uberon), the Human Disease Ontology (DOID) and the Medical Subject Headings (MeSH).

XMap and Lily rely on a subset of synonyms provided by the UMLS Metathesaurus. Since the largebio reference alignment is also based on the UMLS-Metathesaurus we disallowed the use of this UMLS-based dictionary. XMap still produced competitive results, while Lily produced an empty set of alignments.

Alignment coherence

Together with Precision, Recall, F-measure and Runtimes we have also evaluated the coherence of alignments. We have reported (1) number of unsatisfiabilities when reasoning with the input ontologies together with the computed mappings, and (2) the ratio/degree of unsatisfiable classes with respect to the size of the union of the input ontologies.

We have used the OWL 2 reasoner HermiT to compute the number of unsatisfiable classes. For the cases in which HermiT could not cope with the input ontologies and the mappings (in less than 2 hours) we have provided a lower bound on the number of unsatisfiable classes (indicated by ≥) using the OWL 2 EL reasoner ELK.

As in previous OAEI editions, only three systems have shown mapping repair facilities, namely: AML, LogMap (including its LogMapBio variant) and XMap (uses Alcomo repair system). All three systems produce relatively clean outputs in FMA-NCI and FMA-SNOMED cases; however in the SNOMED-NCI cases AML and XMap mappings lead to a number of unsatisfiable classes. XMap seems to deactivate the repair facility for the SNOMED-NCI case. The results also show that even the most precise alignment sets may lead to a large amount of unsatisfiable classes. This proves the importance of using techniques to assess the coherence of the generated alignments.


1. System runtimes and task completion

System FMA-NCI FMA-SNOMED SNOMED-NCI Average # Tasks
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6
LogMapLt 1 6 1 9 5 11 6 6
DOME 2 12 2 20 10 24 12 6
AML 24 55 68 94 346 168 126 6
XMap 7 65 26 299 124 427 158 6
LogMap 6 51 33 287 123 475 163 6
FCAMapX 40 881 91 1,736 833 2,377 993 6
LogMapBio 701 1,072 890 1,840 1,500 2,942 1,491 6
POMAP++ 254 - 779 - 6,312 - 2,448 3
ALOD2Vec 342 - 1,400 - 9,209 - 3,650 3
KEPLER 588 - 4,163 - - - 2,376 2
# Systems 10 7 10 7 9 7 1,142 50
Table 1: System runtimes (s) and task completion.


2. Results for the FMA-NCI matching problem

Task 1: FMA-NCI small fragments

<
System Time (s) # Mappings # Unique Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 24 2,723 77 0.958 0.910 0.933 2 0.020%
FCAMapX 40 2,828 89 0.948 0.911 0.929 4,703 46.2%
LogMapBio 701 2,776 14 0.941 0.902 0.921 2 0.020%
LogMap 6 2,747 3 0.944 0.897 0.920 2 0.020%
ALOD2Vec 342 2,528 6 0.972 0.839 0.901 2,060 20.2%
KEPLER 588 2,506 42 0.960 0.831 0.891 3,707 36.4%
POMAP++ 254 2,414 9 0.979 0.814 0.889 993 9.751%
LogMapLt 1 2,480 11 0.967 0.819 0.887 2,104 20.7%
XMap 7 2,315 2 0.977 0.783 0.869 2 0.020%
DOME 2 2,248 1 0.985 0.764 0.861 324 3.181%
Table 2: Results for the largebio task 1.

Task 2: FMA-NCI whole ontologies

System Time (s) # Mappings # Unique Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 55 2,968 311 0.838 0.872 0.855 10 0.007%
LogMap 51 2,701 0 0.856 0.808 0.831 9 0.006%
LogMapBio 1,072 2,860 39 0.830 0.831 0.830 9 0.006%
XMap 65 2,415 52 0.878 0.742 0.804 9 0.006%
FCAMapX 881 3,607 443 0.665 0.841 0.743 84,601 58.1%
LogMapLt 6 3,458 250 0.676 0.819 0.741 26,267 18.0%
DOME 12 2,383 10 0.803 0.668 0.729 3,595 2.5%
Table 3: Results for the largebio task 2.


3. Results for the FMA-SNOMED matching problem

Task 3: FMA-SNOMED small fragments

System Time (s) # Mappings # Unique Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
FCAMapX 91 7,582 574 0.955 0.815 0.879 13,680 58.0%
AML 68 6,988 415 0.923 0.762 0.835 0 0.000%
LogMapBio 890 6,319 8 0.947 0.693 0.800 1 0.004%
LogMap 33 6,282 1 0.947 0.690 0.798 1 0.004%
XMap 26 5,815 36 0.962 0.647 0.774 0 0.000%
KEPLER 4,163 4,005 375 0.822 0.424 0.559 3,335 14.1%
POMAP++ 779 2,163 49 0.906 0.260 0.404 894 3.793%
ALOD2Vec 1,400 1,727 47 0.941 0.213 0.347 818 3.471%
LogMapLt 1 1,642 2 0.968 0.208 0.342 771 3.271%
DOME 2 1,530 0 0.988 0.198 0.330 751 3.186%
Table 4: Results for the largebio task 3.

Task 4: FMA whole ontology with SNOMED large fragment

System Time (s) # Mappings # Unique Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
FCAMapX 1,736 7,971 1,258 0.819 0.762 0.789 76,120 37.8%
AML 94 6,571 462 0.882 0.687 0.772 0 0%
LogMapBio 1,840 6,471 31 0.834 0.650 0.731 0 0%
LogMap 287 6,393 0 0.840 0.645 0.730 0 0%
XMap 299 6,749 1,217 0.723 0.608 0.661 0 0%
LogMapLt 9 1,820 56 0.851 0.208 0.334 4,389 2.2%
DOME 20 1,588 1 0.941 0.197 0.326 4,314 2.1%
Table 5: Results for the largebio task 4.


4. Results for the SNOMED-NCI matching problem

Task 5: SNOMED-NCI small fragments

System Time (s) # Mappings # Unique Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 346 14,435 1,699 0.878 0.737 0.801 ≥3,244 ≥4.3%
FCAMapX 833 13,789 704 0.878 0.703 0.781 ≥62,854 ≥83.7%
LogMapBio 1,500 12,678 139 0.912 0.672 0.774 ≥0 ≥0%
LogMap 123 12,414 0 0.922 0.665 0.773 ≥0 ≥0%
LogMapLt 5 10,921 65 0.893 0.566 0.693 ≥60,447 ≥80.5%
XMap 124 12,125 1,114 0.835 0.588 0.690 ≥49,878 ≥66.4%
POMAP++ 6,312 10,895 356 0.889 0.563 0.689 ≥57,713 ≥76.9%
DOME 10 9,321 8 0.922 0.499 0.648 ≥44,232 ≥58.9%
ALOD2Vec 9,209 12,882 2,180 0.743 0.556 0.636 ≥49,480 ≥65.9%
Table 6: Results for the largebio task 5.


Task 6: NCI whole ontology with SNOMED large fragment

System Time (s) # Mappings # Unique Scores Incoherence Analysis
Precision  Recall  F-measure Unsat. Degree
AML 168 13,176 1,230 0.904 0.668 0.768 ≥720 ≥0.4%
FCAMapX 2,377 15,383 1,670 0.796 0.680 0.733 ≥151,001 ≥79.8%
LogMapBio 2,942 13,098 231 0.854 0.627 0.723 ≥6 ≥0.003%
LogMap 475 12,276 0 0.867 0.596 0.706 ≥4 ≥0.002%
LogMapLt 11 12,864 720 0.798 0.566 0.662 ≥150,454 ≥79.5%
DOME 24 9,702 42 0.907 0.485 0.632 ≥101,436 ≥53.6%
XMap 427 16,271 4,432 0.640 0.582 0.610 ≥157,345 ≥83.2%
Table 7: Results for the largebio task 6.