Siddhesh Sali

← All work

ml

Language Guidance for Deep Metric Learning

Using pretrained language models as a supervisory signal for vision embeddings — a multi-stage schedule that fine-tunes intermediate layers rather than only the head.

A diagram: on the left three classes of points overlap in one cloud; on the right the same three are pulled into separate clusters, joined to the left by fine curves.

Deep metric learning trains a network to put similar images close together in an embedding space. This project explored how language supervision could add semantic structure to embeddings learned from images.

The opening is that ImageNet pretraining gives you pseudolabels — and a pseudolabel is a word. Once you have words you can reach for a pretrained language model, which knows that a chihuahua and a terrier are both dogs, and that a dog is not a bus. That semantic structure is free, needs no extra annotation, and is exactly the structure the embedding space is trying to learn.

My thesis work was on where that guidance should be applied. The usual move is to supervise the final embedding. Instead this fine-tunes intermediate layers against the language signal across several training stages, so the semantics shape the representation as it forms rather than being imposed on it at the end.

The premise reproduces cleanly. On CUB-200 with a ResNet-50 backbone and a 128-dimensional embedding, both language-guided runs sit well above the plain multi-similarity baseline — that gap is the published result, and it is real. What I can show of my own contribution is training behaviour: swapping the order of the hint and distillation stages moves the loss curve, either side of the published run — which is evidence that where the guidance lands matters, and not yet evidence about what it is worth.

Credit where it is due

This built on published work — Roth et al., Integrating Language Guidance into Vision-based Deep Metric Learning (CVPR 2022) — and my repository is a fork of theirs. The base pipeline, the datasets and much of the training scaffolding are the authors’. My contribution was the hint learning and multi-stage training schedule described above.

Supervised by Prof. Biplab Banerjee at IIT Bombay, July – December 2022. It ran straight on into out-of-distribution detection, which was the second half of the same question.

Gallery

0303
A diagram: a batch of bird photographs runs through an ImageNet-pretrained backbone into an embedding and a similarity matrix; below it, the same photograph produces top-K class words that a language model turns into a second similarity matrix, and a hint loss joins the two.
Fig. 01Hint learning. The pseudolabels are free, the language model is pretrained, and the loss between the two similarity matrices is the whole idea.
A plot of test F1 against training step for three runs on CUB-200: a plain multi-similarity baseline at about 0.37, and two language-guided runs at about 0.41 and 0.42.
Fig. 02On CUB-200 with a 128-dimensional embedding: the two language-guided runs sit clearly above the multi-similarity baseline.
A plot of training loss against step for three runs, all falling from about 1.0 to roughly 0.74, with the two hint-learning orderings finishing either side of the published run.
Fig. 03Two orderings of the hint and distillation stages, against the published run between them. Training loss only — where a schedule shows up first, not where it is judged.