smolagents Vision Support Update

Hugging Face has integrated vision support into smolagents, allowing Vision Language Models (VLMs) to be used natively within agentic pipelines. This update enables agents to process visual information, overcoming the "vision wall" where critical data—such as object positioning, color-coded messages, and icons—is lost during text-only extraction.

Native Vision Integration Methods

smolagents provides two primary ways to pass images to an agent, depending on whether the visual data is static or dynamic.

Static Image Input at Initialization

For use cases like Document AI or analyzing long PDFs with visual elements, images can be passed once at the start of a task. This is achieved by providing a list of images to the run method:

agent.run("Describe these images:", images=[image_1, image_2])

These inputs are stored in the task_images attribute of the TaskStep and are passed to the model alongside the prompt.

Dynamic Image Input via Callbacks

For environments where the visual state changes—such as a web browser—images must be added dynamically to the agent's memory. smolagents achieves this through the MultiStepAgent class and its ReAct framework loop.

At the end of each step in the ReAct loop, the agent executes all functions defined in agent.step_callbacks. By creating a custom callback, developers can log new images into the observation_images attribute of the ActionStep in the agent's memory. This allows the model to see the immediate impact of its previous action before deciding on the next step.

Building a Vision-Enabled Web Browser Agent

Hugging Face demonstrates the utility of these updates by creating an autonomous web browsing agent using the helium library (built on selenium) and CodeAgent.

Technical Implementation

To implement a browsing agent with vision, the following components are utilized:

  • Custom Tools: Specialized tools are created for actions that are difficult for automation libraries to handle, such as go_back() for navigation and close_popups() to dismiss modals using CSS selectors.
  • Vision Callback: A save_screenshot callback is implemented to capture the browser's current state as a PNG, convert it to a PIL image, and assign it to step_log.observations_images at the end of every step.
  • Model Configuration: Support for images is available across all models. When using TransformersModel with a VLM, the flatten_messages_as_text parameter must be set to False during initialization to ensure images are processed correctly.

Example Workflow

In a provided example, a CodeAgent is tasked with finding the total commits of the top author of the top trending GitHub repository. The agent uses a combination of helium for page interaction, custom tools for navigation, and a VLM (such as Qwen2VL-72B) to visually interpret the page layout and navigate to the correct profile.

Model Compatibility and Performance

Vision support is integrated across the smolagents model suite. The effectiveness of vision-based tasks depends heavily on the strength of the VLM used. Hugging Face notes that high-capacity models like Qwen2VL-72B or GPT-4o demonstrate higher success rates for complex visual navigation tasks.

Sources