Hugging Face AI Sheets Vision Update

Hugging Face has released a significant update to AI Sheets, an open-source tool designed for building, transforming, and enriching datasets using open AI models without requiring code. The primary addition is comprehensive vision support, enabling users to analyze, extract data from, and generate visual content directly within a spreadsheet workflow.

Vision-Based Data Extraction and Analysis

AI Sheets allows users to upload images or use image-based datasets to extract and structure information using vision models. This capability transforms unstructured visual data—such as receipts, product catalogs, and scanned documents—into organized tables.

Key capabilities include:

  • Structured Data Extraction: Users can pull specific line items from receipts, data from charts, or text from scanned documents using custom prompts.
  • Image Description and Categorization: The tool can generate captions for photos, classify document types, or tag images based on content.
  • Metadata Generation: Models can automatically label images with quality scores, relevant attributes, or custom annotations.

Users can refine these outputs by iterating on prompts, manually editing results, and using a "thumbs-up" mechanism to provide few-shot examples to the model for improved accuracy.

Integrated Image Generation and Editing

Beyond extraction, AI Sheets integrates image-to-image and text-to-image models to facilitate content creation within the spreadsheet.

  • Text-to-Image Generation: Users can create social media graphics, thumbnails, or illustrations based on text columns (e.g., generating a food photo based on a recipe title).
  • Image Transformation: Existing images can be modified to change styles, add elements, or adjust compositions.
  • Image Variations: The tool supports creating multiple versions of a visual at scale for testing and branding purposes.

Technical Implementation and Model Flexibility

AI Sheets leverages Hugging Face Inference Providers, granting users access to thousands of open models. This allows users to swap models based on the specific needs of their task, balancing speed and accuracy.

In a provided example of extracting text from handwritten recipes, the tool demonstrated the difference between model tiers:

  • Qwen/Qwen2.5-VL-7B-Instruct: Used as a default for a balance of speed and accuracy.
  • Qwen/Qwen3-VL-235B-A22B-Reasoning: A state-of-the-art reasoning model that proved more accurate in detecting subtle details, such as specific ingredients (e.g., "spinach") and precise cooking times (e.g., "50-60 min"), which the smaller model missed.

Workflow and Export Options

The AI Sheets workflow follows a linear path from data ingestion to final export:

  1. Upload: Users upload folders of images or connect to datasets.
  2. AI Actions: Users apply operations to columns. Image columns support text extraction, object detection, and colorization, while text columns support summarization, translation, and keyword extraction.
  3. Enrichment: Extracted text can be further processed (e.g., converting raw transcription into structured HTML).
  4. Export: The final dataset can be exported to the Hugging Face Hub (publicly or privately) in CSV or Parquet formats.

AI Sheets is available as a hosted Space for immediate use or can be deployed locally via its GitHub repository.

Sources