Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings

Abstract

Human sketch has already proved its worth in various visual understanding tasks (e.g., retrieval, segmentation, image-captioning, etc). In this paper, we reveal a new trait of sketches - that they are also salient. This is intuitive as sketching is a natural attentive process at its core. More specifically, we aim to study how sketches can be used as a weak label to detect salient objects present in an image. To this end, we propose a novel method that emphasises on how "salient object" could be explained by hand-drawn sketches. To accomplish this, we introduce a photo-to-sketch generation model that aims to generate sequential sketch coordinates corresponding to a given visual photo through a 2D attention mechanism. Attention maps accumulated across the time steps give rise to salient regions in the process. Extensive quantitative and qualitative experiments prove our hypothesis and delineate how our sketch-based saliency detection model gives a competitive performance compared to the state-of-the-art.

Architecture

Illustration of photo-to-sketch generation process to learn image saliency from sketch labels. Attention maps accumulated across the time steps give rise to the saliency map.

Example photo-sketch pairs with original, absolute coordinate rasterised and scale normalised photos being at the top, middle and bottom respectively. It is evident that photo and sketches are not aligned even after scale normalisation, as sketch is not a pixel-wise tracing of photo edge map. This invalidates the use of classical template matching algorithms for our problem.

Visualisation of raw attention-map obtained via sketch as a weak label for saliency detection. While most existing weakly supervised methods involve heuristic-based post-processing techniques, all results shown here are directly predicted by the network without any post-processing.

Results

Qualitative results on weakly supervised saliency detection using (a) class-label + sketch (b) class-label + text-caption (c) only sketch (d) only text-caption (e) only class-label. Use of sketch over text-caption significantly improves the quality of saliency map.

BibTeX

@inproceedings{bhunia2023sketch2saliency,
author = {Ayan Kumar Bhunia and Subhadeep Koley and Amandeep Kumar and Aneeshan Sain and Pinaki Nath Chowdhury and Tao Xiang and Yi-Zhe Song},
title = {{Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings}},
booktitle = {CVPR},
year = {2023}}

Sketch2Saliency:

Learning to Detect Salient Objects from Human Drawings

Sequential photo-to-sketch generation with 2D-attention to leverage sketch as a weak label for salient object detection. Aggregated 2D attention-maps till a particular instant are shown.