Kakao Brain ViT and ALIGN Models Release with COYO 700M Dataset

TL;DR

Kakao Brain and Hugging Face have open‑sourced two visual‑language models—ViT and ALIGN—trained on the new 700 M image‑text COYO dataset, marking the first publicly released ALIGN model and the first ViT/ALIGN models paired with an open training corpus.

What was released

  • COYO dataset – 700 M image‑text pairs collected from the web, released under an open‑source license.
  • ViT models – Vision Transformers that follow the architecture and hyper‑parameters of Google’s ViT, trained on the COYO‑Labeled‑300M subset.
  • ALIGN models – Dual‑encoder image‑text models matching Google’s ALIGN architecture, trained on the full COYO dataset.
  • Demo spaces and pipelines – Interactive demos on the Hugging Face Hub and ready‑to‑use transformers pipelines for classification and zero‑shot tasks.

Performance comparison

  • ALIGN‑B7‑Base trained on 700 M pairs matches Google’s ALIGN‑B7‑Base on Image KNN classification and exceeds it on MS‑COCO image‑to‑text and text‑to‑image retrieval.
  • ViT‑L/16 achieves comparable ImageNet and ImageNet‑ReaL accuracy to Google’s ViT‑L/16 at 384 px and 512 px resolutions.
  • These results demonstrate that open‑source models can reach state‑of‑the‑art performance even with a fraction of the proprietary data size.

COYO dataset details

  • Scale: 700 M English image‑text pairs (≈747 M after filtering).
  • Source: Webpages crawled between Oct 2020 and Aug 2021 (Common Crawl).
  • Metadata: Includes CLIP similarity scores (ViT‑B/32 and ViT‑L/14), NSFW scores, watermark scores, aesthetic scores, and face‑count data.
  • Differences from LAION‑2B:
    Feature COYO LAION‑2B
    Size 700 M 2 B
    Similarity score CLIP‑B/32 & L/14, no filtering CLIP‑B/32, threshold 0.28
    NSFW filtering Both image & text Image only
    Face count Provided Not provided
    Watermark score Robust metric Basic score
    Availability Hugging Face Hub Hugging Face Hub

How ViT works

  • ViT splits an image into fixed‑size patches, embeds each patch, adds positional embeddings, and processes the sequence with a standard Transformer encoder.
  • This design yields up to four‑times better compute efficiency than comparable CNNs while remaining domain‑agnostic.
  • Kakao Brain’s ViT models use the same architecture as Google’s ViT but are trained on the publicly released COYO‑Labeled‑300M subset, enabling full reproducibility.

How ALIGN works

  • ALIGN employs a dual‑encoder: a vision encoder for images and a text encoder for captions, trained with a contrastive loss on noisy alt‑text pairs.
  • The noisy, large‑scale training corpus (originally 1.8 B pairs) allows ALIGN to excel at cross‑modal retrieval and zero‑shot classification.
  • Kakao Brain’s ALIGN model is the first open‑source implementation of this architecture, trained on the 700 M COYO pairs and achieving performance that meets or exceeds Google’s reported numbers.

Using the COYO dataset

from datasets import load_dataset
# Load the full dataset (may be large)
full = load_dataset('kakaobrain/coyo-700m')
# Stream a subset to avoid downloading everything
stream = load_dataset('kakaobrain/coyo-700m', streaming=True)
print(next(iter(stream['train'])))

The streamed example shows fields such as url, text, width, height, clip_similarity_vitb32, nsfw_score_opennsfw2, watermark_score, and aesthetic_score_laion_v2.

Quick start with ViT

import requests, torch
from PIL import Image
from transformers import ViTImageProcessor, ViTForImageClassification

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
processor = ViTImageProcessor.from_pretrained('kakaobrain/vit-large-patch16-384')
model = ViTForImageClassification.from_pretrained('kakaobrain/vit-large-patch16-384')

inputs = processor(images=image, return_tensors='pt')
with torch.no_grad():
    logits = model(**inputs).logits
probs = torch.nn.functional.softmax(logits, dim=-1)
top5 = torch.argsort(probs, descending=True)[0, :5]
for idx in top5:
    print(f"{model.config.id2label[idx.item()]}: {probs[0, idx].item():.4f}")

Or use the high‑level pipeline:

from transformers import pipeline
classifier = pipeline('image-classification', model='kakaobrain/vit-large-patch16-384')
print(classifier('http://images.cocodataset.org/val2017/000000039769.jpg', top_k=5))

Quick start with ALIGN

from transformers import AlignProcessor, AlignModel
import requests, torch
from PIL import Image

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
processor = AlignProcessor.from_pretrained('kakaobrain/align-base')
model = AlignModel.from_pretrained('kakaobrain/align-base')

candidate_labels = ['an image of a cat', 'an image of a dog']
inputs = processor(images=image, text=candidate_labels, return_tensors='pt')
with torch.no_grad():
    logits = model(**inputs).logits_per_image
probs = logits.softmax(dim=1)
print(probs)

For zero‑shot classification via pipeline:

from transformers import pipeline
classifier = pipeline('zero-shot-image-classification', model='kakaobrain/align-base')
print(classifier('https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png',
                candidate_labels=['animals', 'humans', 'landscape']))

The model also exposes get_image_features and get_text_features for downstream embedding‑based tasks.

Implications for the research community

  • Reproducibility: Researchers can now replicate Google‑scale ViT and ALIGN experiments because both the models and the exact training data are publicly available.
  • Accessibility: The open‑source ALIGN model removes a major barrier for labs without access to proprietary billions‑of‑pair datasets.
  • Benchmarking: Since COYO includes rich metadata (aesthetic, watermark, face counts), new fine‑grained analyses of model behavior on filtered subsets become possible.
  • Future work: The community can extend COYO, combine it with other datasets (e.g., LAION), or fine‑tune the released models for specialized domains while retaining full transparency.

All code snippets assume a recent version of transformers (or the development branch for ALIGN) and the datasets library installed via pip install datasets.

Sources