apple-aiml-research/ml-flextok

What it solves

FlexTok addresses the challenge of representing images as 1D token sequences of flexible length. Unlike traditional fixed-length tokenization, it allows for the resampling of images into sequences that can be truncated or adjusted without losing the essential image information required for reconstruction.

How it works

FlexTok uses an encoder-decoder architecture. The encoder converts images into discrete token sequences. The decoder, which is a rectified flow decoder, uses these tokens to reconstruct the image. A key feature is that these token sequences can be truncated (e.g., keeping only the first 64 out of 256 tokens) and the model can still perform detokenization to reconstruct the image from a shortened sequence.

Who it’s for

This project is for researchers and developers working on multimodal AI, image tokenization, and generative models who need a flexible way to represent images as sequences for use in LLMs or other sequence-based models.

Highlights

• Flexible Token Length: Ability to truncate token sequences in a nested fashion while maintaining reconstruction capability.
• Rectified Flow Decoder: Employs a rectified flow decoder for high-quality image reconstruction from tokens.
• Pre-trained Models: Provides a variety of FlexTok tokenizers and VAEs (Variational Autoencoders) with different configurations (e.g., different encoder/decoder layers) and latent channels.
• Hugging Face Integration: Easy loading of pre-trained checkpoints via the Hugging Face Hub.