A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch
 
Patsorn Sangkloy
 
Wittawat Jitkrittum
 
Diyi Yang
 
James Hays
Abstract
We address the problem of retrieving in-the-wild images with both a sketch and a text query. We present TASK-former (Text And SKetch transformer), an end-to-end trainable model for image retrieval using a text description and a sketch as input. We argue that both input modalities complement each other in a manner that cannot be achieved easily by either one alone. TASK-former follows the late-fusion dual-encoder approach, similar to CLIP, which allows efficient and scalable retrieval since the retrieval set can be indexed independently of the queries. We empirically demonstrate that using an input sketch (even a poorly drawn one) in addition to text considerably increases retrieval recall compared to traditional text-based image retrieval. To evaluate our approach, we collect 5,000 hand-drawn sketches for images in the test set of the COCO dataset.
Citation
@article{
 tsbir2022,
 author = {Patsorn Sangkloy and Wittawat Jitkrittum and Diyi Yang and James Hays},
 title = {A Sketch is Worth a Thousand Words: Image Retrieval with Text and Sketch},
 journal = {European Conference on Computer Vision, ECCV},
 year = {2022},
}
            
View Paper
Data
Dataset : [ Sketches ] [ Annotation and Info ] [ Sketches in SVG format ]
Code and Network model : [ Github ]

Supplementary Material