Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

Qwen has introduced Chinese CLIP, a contrastive vision-language pretraining model designed to bridge the gap between vision and language specifically for the Chinese language. This model addresses the limitations of existing multimodal models that rely on machine translation or multilingual CLIP variants, which often struggle with Chinese-specific cultural concepts and native data distributions.

Why a Language-Specific CLIP is Necessary

Standard CLIP models and their multilingual counterparts often fail to capture the nuances of Chinese culture and social landscapes. Qwen identifies two primary failure points in existing approaches:

  • Translation Degradation: Using original CLIP combined with machine translation for Chinese text significantly degrades performance compared to native Chinese pretraining.
  • Cultural Bias: Multilingual models like mCLIP often struggle to understand Chinese-specific concepts, frequently retrieving items associated with Western culture instead of native Chinese contexts.

Two-Stage Pretraining Methodology

To efficiently transfer CLIP's capabilities to Chinese, Qwen utilizes a two-stage pretraining method rather than training from scratch, which the researchers describe as a more cost-effective approach.

Stage 1: Language Encoder Tuning

In the first stage, the model initializes two towers: a vision encoder (e.g., ViT-B or ResNet) from the original CLIP and a Chinese RoBERTa (RoBERTA-wwm-Chinese) as the language encoder. The vision encoder is frozen, and the language encoder is contrastively tuned to map its representations into the output space of the CLIP vision encoder.

Stage 2: Joint Fine-Tuning

In the second stage, the vision encoder is unlocked. Both the vision and language encoders are contrastively tuned together. This allows the vision encoder to adapt to the distribution of images found in Chinese data, capturing regional cultural values and visual characteristics.

Training Data and Model Variants

The model was pretrained on approximately 200 million image-text pairs. To ensure reproducibility, the team primarily used public datasets, including:

  • The Chinese ("zh") portion of LAION-5B
  • The Wukong dataset
  • Translated data from MSCOCO and Visual Genome

Qwen released five model variants to support different performance and efficiency needs:

  • ResNet-50
  • ViT-B/16
  • ViT-L/14
  • ViT-L/14 @336px
  • ViT-H/14

Performance and Benchmarks

Chinese CLIP demonstrates superior performance across multiple retrieval and classification tasks, particularly on native Chinese datasets.

Cross-Modal Retrieval

Experiments were conducted on three datasets: MUGE (a Chinese native dataset), Flickr30K-CN, and COCO-CN (English-native datasets). Chinese CLIP outperformed previous best models on all three, with the most significant performance gap observed on the MUGE dataset, indicating a strong capability to handle native Chinese data.

Zero-Shot Image Classification

Using the ELEVATER benchmark with manually translated labels and prompts, Chinese CLIP achieved competitive performance, demonstrating its effectiveness even on English-native benchmarks.

Ablation Results

Ablation studies confirmed that the two-stage pretraining approach is significantly more effective than training from scratch, and that the second stage of tuning further enhances cross-modal retrieval performance.

Future Directions

While Chinese CLIP is effective for retrieval and classification, the researchers aim to further validate its role as a vision foundation model. Future work will focus on building dedicated benchmarks for Chinese multimodal and vision representation learning.

Sources