Ontology Alignment Evaluation Initiative - OAEI-2012 Campaign

Results OAEI 2012::Large BioMed Track

Results OAEI 2012 FMA-NCI matching problem

FMA-NCI small fragments

This year we obtained very high level participation and 11 systems/configurations obtained, on average, an F-measure over 0.80 for the matching problem involving the small fragments of FMA and NCI. GOMMA-bk obtained the best results in terms of both recall and F-measure while ServOMap provided the most precise alignments. LogMap and LogMap-noe provided the same results since the input ontologies are already small fragments of FMA and NCI and thus, the overlapping estimation performed by LogMap did not have any impact. In general, as expected, precision increases when comparing against the original UMLS mapping set, while recall decreases.

Our baseline provided very good results in terms of F-measure and outperformed 8 of the participating systems. MaasMatch and Hertuda provided competitive results in terms of recall, but the low precision damaged the final F-measure. MapSSS and AUTOMSv2 provided a set of mappings with high precision, however, the F-measure was damaged due to the low recall of their mappings.

The runtimes were very positive in general and 8 systems completed the task in less than 2 minutes. MapSSS required less than 10 minutes, while Hertuda and HotMatch required around 1 hour. Finally, MaasMatch, AUTOMSv2 and Wmatch needed 8, 17 and 18 hours to complete the task, respectively.

Regarding mapping coherence, only LogMap (with its two variants) generates an almost clean output. In the table, we can appreciate that even the most precise mappings (ServOMap or YAM++) lead to a huge amount of unsatisfiable classes when reasoning together with the input ontologies; and thus, it proves the importance of using techniques to assess the coherence of the generated alignments. Unfortunately, LogMap and CODI are the unique systems (participating in the OAEI 2012) that have shown to use such techniques.


System Time (s) # Mappings Original UMLS Refined UMLS (LogMap) Refined UMLS (Alcomo) Average Incoherence Analysis
Precision  Recall  F-measure Precision  Recall  F-measure Precision  Recall  F-measure Precision  Recall  F-measure All Unsat. Degree Root Unsat.
GOMMA-Bk 26 2,843 0.961 0.903 0.931 0.932 0.914 0.923 0.914 0.922 0.918 0.936 0.913 0.924 6,204 60.92% 193
YAM++ 78 2,614 0.980 0.848 0.909 0.959 0.865 0.910 0.933 0.866 0.898 0.958 0.859 0.906 2,352 23.10% 92
LogMap/LogMap-noe 18 2,740 0.952 0.863 0.905 0.934 0.883 0.908 0.908 0.883 0.895 0.932 0.876 0.903 2 0,02% 0
GOMMA 26 2,626 0.973 0.845 0.904 0.945 0.856 0.898 0.928 0.865 0.896 0.949 0.855 0.900 2,130 20.92% 127
ServOMapL 20 2,468 0.988 0.806 0.888 0.964 0.821 0.887 0.936 0.819 0.873 0.962 0.815 0.883 5,778 56.74% 79
LogMapLt 8 2,483 0.969 0.796 0.874 0.942 0.807 0.869 0.924 0.814 0.866 0.945 0.806 0.870 2,104 20.66% 116
ServOMap 25 2,300 0.990 0.753 0.855 0.969 0.769 0.857 0.949 0.774 0.853 0.969 0.765 0.855 5,597 54.96% 50
HotMatch 4,271 2,280 0.971 0.732 0.835 0.951 0.748 0.838 0.947 0.766 0.847 0.957 0.749 0.840 285 2.78% 65
Wmatch 65,399 3,178 0.811 0.852 0.831 0.786 0.862 0.823 0.767 0.864 0.813 0.788 0.860 0.822 3,168 31.11% 482
AROMA 63 2,571 0.876 0.745 0.805 0.854 0.758 0.803 0.837 0.764 0.799 0.856 0.756 0.803 7,196 70.66% 421
Hertuda 3,327 4,309 0.598 0.852 0.703 0.578 0.860 0.691 0.564 0.862 0.682 0.580 0.858 0.692 2,675 26.27% 277
MaasMatch 27,157 3,696 0.622 0.765 0.686 0.606 0.778 0.681 0.597 0.788 0.679 0.608 0.777 0.682 9,598 94.25% 3,113
AUTOMSv2 62,407 1,809 0.821 0.491 0.615 0.802 0.501 0.617 0.709 0.507 0.618 0.804 0.500 0.616 5,346 52.49% 392
MapSSS 561 1,483 0.860 0.422 0.566 0.840 0.430 0.568 0.829 0.436 0.571 0.843 0.429 0.569 565 5.55% 94


FMA-NCI big fragments

AUTOMSv2, HotMatch, Hertuda, Wmatch and MaasMatch failed to complete the task involving the big fragments of FMA and NCI after more than 24 hours of execution. Runtimes were in line with the small matching task, apart from the ones for MapSSS and AROMA which suffered an important increase.

YAM++ provided the best results in terms of F-measure, whereas GOMMA-bk and ServOMap got the best recall and precision, respectively. F-measures have decreased considerably with respect to the small matching task. This is mostly due to the fact that this matching task involves more possible candidate mappings than the previous one. Nevertheless, seven systems outperformed our baseline and provided high quality mapping sets in terms of both precision and recall. Only, MapSSS and AROMA provided worse results in terms of both precision and recall than LogMapLt.

Regarding mapping coherence, as in the previous task, only LogMap (with its two variants) generates an almost clean output where the mappings together with the input ontologies only lead to 5 unsatisfiable classes.


System Time (s) # Mappings Original UMLS Refined UMLS (LogMap) Refined UMLS (Alcomo) Average Incoherence Analysis
Precision  Recall  F-measure Precision  Recall  F-measure Precision  Recall  F-measure Precision  Recall  F-measure All Unsat. Degree Root Unsat.
YAM++ 245 2,688 0.923 0.821 0.869 0.904 0.838 0.870 0.878 0.838 0.857 0.902 0.832 0.866 22,402 35.49% 102
ServOMapL 95 2,640 0.914 0.798 0.852 0.892 0.812 0.850 0.866 0.811 0.838 0.891 0.807 0.847 22,315 35.41% 143
GOMMA 69 2,810 0.876 0.814 0.844 0.856 0.830 0.843 0.840 0.837 0.838 0.857 0.827 0.842 2,398 4.40% 116
GOMMA_Bk 83 3,116 0.832 0.857 0.844 0.814 0.875 0.843 0.796 0.880 0.836 0.814 0.871 0.841 4,609 8.46% 146
LogMap-noe 74 2,663 0.888 0.782 0.832 0.881 0.809 0.843 0.848 0.801 0.824 0.872 0.798 0.833 5 0.01% 0
LogMap 77 2,656 0.887 0.779 0.829 0.877 0.803 0.838 0.846 0.797 0.821 0.870 0.793 0.830 5 0.01% 0
ServOMap 98 2,413 0.933 0.744 0.828 0.913 0.760 0.829 0.894 0.766 0.825 0.913 0.757 0.828 21,688 34.03% 86
LogMapLt 29 3,219 0.748 0.796 0.771 0.726 0.807 0.764 0.713 0.814 0.760 0.729 0.806 0.766 12,682 23.29% 443
AROMA 7,538 3,856 0.541 0.689 0.606 0.526 0.700 0.601 0.514 0.703 0.594 0.527 0.698 0.600 20,054 24.07% 1600
MapSSS 30,575 2,584 0.392 0.335 0.362 0.384 0.342 0.362 0.377 0.345 0.360 0.384 0.341 0.361 21,893 40.21% 358
HotMatch - - - - - - - - - - - - - - - - -
Wmatch - - - - - - - - - - - - - - - - -
Hertuda - - - - - - - - - - - - - - - - -
MaasMatch - - - - - - - - - - - - - - - - -
AUTOMSv2 - - - - - - - - - - - - - - - - -


FMA-NCI whole ontologies

AROMA and MapSSS failed to complete the matching task involving the whole FMA and NCI ontologies in less than 24 hours.

As in the previous task, the remaining 7 matching systems generated high quality mapping sets. YAM++ provided the best results in terms of F-measure, whereas GOMMA-bk and ServOMap got the best recall and precision, respectively. LogMap with its two configurations provided an almost clean output and only 9 classes where unsatisfiable after reasoning with the input ontologies and the computed mappings.

Runtimes were also very positive. YAM++ was slightly slower than the other systems, which gave the outputs in less than 5 minutes, and required around 20 minutes to complete the task.


System Time (s) # Mappings Original UMLS Refined UMLS (LogMap) Refined UMLS (Alcomo) Average Incoherence Analysis
Precision  Recall  F-measure Precision  Recall  F-measure Precision  Recall  F-measure Precision  Recall  F-measure All Unsat. Degree Root Unsat.
YAM++ 1,304 2,738 0.907 0.821 0.862 0.887 0.838 0.862 0.862 0.838 0.850 0.885 0.832 0.858 50,550 28.56% 141
GOMMA 217 2,843 0.865 0.813 0.839 0.846 0.830 0.837 0.829 0.836 0.833 0.847 0.826 0.836 5,574 3.83% 139
ServOMapL 251 2,700 0.891 0.796 0.841 0.869 0.810 0.839 0.844 0.808 0.826 0.868 0.805 0.835 50,334 28.48% 164
GOMMA_Bk 231 3,165 0.818 0.856 0.837 0.800 0.874 0.836 0.783 0.879 0.828 0.801 0.870 0.834 12,939 8.88% 245
LogMap-noe 206 2,646 0.882 0.771 0.823 0.875 0.799 0.835 0.842 0.790 0.815 0.866 0.787 0.825 9 0.01% 0
LogMap 131 2,652 0.875 0.768 0.818 0.868 0.795 0.830 0.836 0.786 0.810 0.860 0.783 0.819 9 0.01% 0
ServOMap 204 2,465 0.912 0.743 0.819 0.892 0.759 0.820 0.873 0.764 0.815 0.892 0.755 0.818 48,743 27.31% 114
LogMapLt 55 3,466 0.695 0.796 0.742 0.675 0.807 0.735 0.662 0.814 0.730 0.677 0.806 0.736 26,429 8.68% 778
AROMA - - - - - - - - - - - - - - - - -
MapSSS - - - - - - - - - - - - - - - - -
HotMatch - - - - - - - - - - - - - - - - -
Wmatch - - - - - - - - - - - - - - - - -
Hertuda - - - - - - - - - - - - - - - - -
MaasMatch - - - - - - - - - - - - - - - - -
AUTOMSv2 - - - - - - - - - - - - - - - - -