Identifying Pitfalls in the Evaluation of Saliency Models for Videos

Saliency prediction has been extensively studied for natural images. In the area of video coding and video quality assessment, researchers attempt to integrate a saliency model to individual frames of a video sequence. In selecting best-performing saliency models for these applications, the evaluation only considers the average model performance over all frames of a video. This may mask the defects of a saliency model and consequently hinder further improvement of the model. In this paper, we present the identification of pitfalls in the evaluation of saliency models for videos. We demonstrate the importance of considering the video content classification and temporal effect. Building on the findings, we make recommendations for saliency model evaluation and selection for videos.