Skip to main content

MvCo−DoT: Multi−View Contrastive Domain Transfer Network for Medical Report Generation

Ruizhi Wang‚ Xiangtao Wang‚ Zhenghua Xu‚ Wenting Xu‚ Junyang Chen and Thomas Lukasiewicz


In clinical scenarios, multiple medical images with different views are usually generated at the same time, and they have high semantic consistency. However, the existing medical report generation methods cannot exploit the rich multi-view mutual information of medical images. Therefore, in this work, we propose the first multi-view medical report generation model, called MvCo-DoT. Specifically, MvCo-DoT first propose a multi-view contrastive learning (MvCo) strategy to help the deep reinforcement learning based model utilize the consistency of multi-view inputs for better model learning. Then, to close the performance gaps of using multi-view and single-view inputs, a domain transfer network is further proposed to ensure MvCo-DoT achieve almost the same performance as multi-view inputs using only single-view inputs. Extensive experiments on the IU X-Ray public dataset show that MvCo-DoT outperforms the SOTA medical report generation baselines in all metrics.

Book Title
Proceedings of the IEEE International Conference on Acoustics‚ Speech and Signal Processing‚ ICASSP 2023‚ Rhodes Island‚ Greece‚ 4−10 June 2023