Training-Free Sketch-Based Image Retrieval Based on Multimodal Feature Fusion and Image Generation

Abstract

Sketch-Based Image Retrieval (SBIR) aims to retrieve relevant photographs from a large database using a freehand sketch as the query. The core challenge is the domain gap between abstract sketches and natural photographs, which causes most existing approaches to depend on task-specific training on labelled sketch–photo pairs, limiting their generalisability across datasets and categories. This thesis proposes a training-free SBIR framework that combines three pretrained models — CLIP, BLIP, and stable diffusion with ControlNet — without any fine-tuning. Two retrieval paradigms are explored and evaluated across four benchmark datasets: Sketchy, Extended Sketchy, TU-Berlin, and QuickDraw. In the direct retrieval paradigm, the query is represented by a weighted combination of CLIP visual features, BLIP-generated captions, and CLIP-predicted class labels, explored across seven fixed ablation configurations and a dataset-specific parameter search. In the generation-based paradigm, the sketch is first converted into a realistic image using stable diffusion conditioned on either a BLIP caption or a CLIP-predicted class label, and the generated image is used as the retrieval query. The key finding is that text-based signals — class labels and captions — consistently outperform raw sketch visual features for category-level retrieval. Because CLIP was not trained on sketch images, text descriptions bridge the domain gap more reliably than visual matching. The optimal feature combination is dataset-dependent: highly abstract datasets such as QuickDraw benefit most from class labels alone, while richer datasets gain from incorporating captions. The generation-based approach consistently underperforms direct retrieval due to the domain shift between generated and real photographs and error propagation through the generation pipeline. The proposed framework outperforms all four evaluated trained state-of-the-art methods on QuickDraw (mAP@All = 0.437 vs. best trained 0.231, +89.2%), achieves the highest mAP@200 on Extended Sketchy Split 2 (0.761), and matches the best trained method on TU-Berlin P@100 within 0.4% — all without any task-specific training, demonstrating that competitive zero-shot SBIR is achievable using pretrained models alone.