Joint Coupled Transform Learning Framework For Multimodal Image Super-Resolution

Insights from multiple imaging modalities have recently been applied in solving many computer vision related applications. In this paper, we model the cross-modal dependencies between different modalities for Multimodal Image Super-Resolution (MISR), i.e., enhance the Low Resolution (LR) image of target modality with the guidance of a High Resolution (HR) image from another modality. We introduce a joint optimization based transform learning framework referred to as Joint Coupled Transform Learning (JCTL) to combine the information from multiple modalities to generate the HR image of the target modality. All the necessary intermediate steps and the corresponding closed form solution updates are provided. The performance of the proposed JCTL is benchmarked against the state-of-the-art MISR approaches on different multimodal datasets with different upscaling factors. The results show better performance with the proposed JCTL approach compared to other state-of-the-art techniques both in terms of PSNR and SSIM.