The fitted "color" direction makes the model answer "A" every time
The fitted "color" direction makes the model answer "A" every time
Share of answers that were "A", same question with three different additions inside the model
Normal, nothing added0.50
Add the fitted direction1.00
Add a random direction, same size0.50
Note: The test set is balanced, so answering "A" every time scores 0.50, chance. A direction that really swapped the two pictures would push accuracy toward 0. Accuracy was 1.00, 0.50 and 1.00 in the three runs.
Source: Taylor et al., NeurIPS 2026 InterpScience workshop. Qwen2.5-VL-3B-Instruct unless noted.