NVIDIA study finds tool-using vision agents refuse harmful requests less often

What is changing?

A study by six NVIDIA researchers, accepted at NeurIPS 2026, finds that multimodal large language models can become less likely to refuse harmful requests when turned into tool-using agents. The team evaluated 11 models from seven families across MM-SafetyBench, VLSBench and HoliSafe, collecting more than 100,000 responses. In experiments using tagging, zoom-and-crop, OCR and sandboxed Python tools, every model tested was less safe in the tool-enabled setting; the relative increase in refusal failures reached 68.7% in some configurations.

The researchers describe two possible mechanisms: long tool traces can dilute the original safety instruction in context, while the model's attention shifts from the user's intent to processing tool outputs. Reintroducing the original request and image after the final tool call reduced failures by an average of 7.6% in the tested subset. The controlled study does not prove that every deployed agent automatically becomes unsafe; it measures how tool orchestration can alter safety behavior.