OpenAI announces GPT‑4 powered Virtual Volunteer for Be My Eyes

TL;DR

OpenAI integrated GPT‑4’s visual‑input capability into the Be My Eyes app, launching a “Virtual Volunteer” that can describe, analyze, and converse about images, giving blind and low‑vision users real‑time, context‑rich assistance.

What was announced

OpenAI announced that the Danish startup Be My Eyes is embedding GPT‑4 (research preview) into its mobile app to power a Virtual Volunteer™. The assistant can process user‑submitted photos, generate detailed descriptions, answer follow‑up questions, and provide actionable advice such as recipe suggestions or navigation instructions.

Why it matters

The Virtual Volunteer delivers a level of visual interpretation and interactive dialogue that surpasses traditional image‑to‑text tools, promising greater independence for the estimated 250 million people worldwide who are blind or have low vision.

Technical foundation

  • GPT‑4 visual input: The model accepts image data in addition to text, enabling multimodal reasoning.
  • Conversational context: Unlike static object‑recognition APIs, GPT‑4 can maintain a back‑and‑forth dialogue, refining its answers based on user prompts.
  • Analytical prowess: The system can not only name objects but also infer relationships, assess safety hazards, and suggest next steps (e.g., “are these noodles suitable for a recipe?”).

Core capabilities

Image description and analysis

The assistant can identify objects, read text within images, and provide nuanced explanations. For example, a photo of a refrigerator yields a list of items, possible recipes, and nutritional information.

Real‑time navigation assistance

A user was able to receive step‑by‑step directions through a railway station, including map positioning and safety warnings, demonstrating the model’s ability to interpret complex spatial information.

Web‑page comprehension

GPT‑4 can be shown an entire webpage, then summarize the most relevant sections, filter out clutter, and present concise answers—helping users browse news sites, e‑commerce pages, and other visually dense content.

Early testing and feedback

  • Beta rollout: In early February 2023, Be My Eyes began internal beta testing with a small group of employees.
  • Positive reception: CEO Michael Buckley called the performance “unparalleled” compared to existing image‑to‑text tools. Beta participants, including accessibility advocate Lucy Edwards, expressed enthusiasm for the new functionality.

“The difference between GPT‑4 and other language and machine learning models is both the ability to have a conversation and the greater degree of analytical prowess offered by the technology.” – Jesper Hvirring Henriksen, CTO, Be My Eyes

Implications for accessibility

  • Increased independence: Users can obtain instant, detailed visual information without waiting for a human volunteer, expanding the range of tasks they can perform alone.
  • Broader application scope: Beyond everyday objects, the technology can assist with reading screens, evaluating product safety, and making purchasing decisions on cluttered e‑commerce sites.
  • Commercial potential: Buckley highlighted the dual impact of societal benefit and a sizable market opportunity for AI‑driven accessibility solutions.

Next steps

OpenAI and Be My Eyes plan to roll the Virtual Volunteer out to the wider Be My Eyes user base within weeks of the beta, followed by continuous refinement based on user feedback and further safety evaluations.

This post summarizes the official OpenAI announcement dated March 14 2023. All statements are drawn directly from the source material.

Sources