bilibili/Index-1.9B

What it solves

Index-1.9B is a series of lightweight small language models (SLMs) designed to provide high-performance text generation, dialogue, and long-context processing while maintaining a small parameter footprint (1.9 billion parameters). It aims to bridge the gap between extremely small models and larger models in terms of capabilities, particularly for Chinese and English languages, and specialized tasks like role-playing and long-document analysis.

How it works

The project provides several model variants based on a base model pre-trained on 2.8 trillion tokens of mainly Chinese and English corpus:

• Index-1.9B base: The foundational model.
• Index-1.9B pure: A control version used to verify the impact of instruction data by filtering out all instruction-related data.
• Index-1.9B chat: A dialogue-aligned model using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO).
• Index-1.9B character: A role-playing model that integrates Retrieval-Augmented Generation (RAG) on top of SFT and DPO for few-shot customization.
• Index-1.9B-32K: A version specifically optimized for long contexts (up to 32,000 tokens) through continue pre-training and SFT on curated long-text data.

Who it’s for

• Developers looking for efficient, lightweight models that can be deployed on devices with limited memory.
• Researchers interested in the impact of instruction data on model performance (via the "pure" model).
• Content creators wanting to build customized role-playing characters.
• Users needing a small model capable of summarizing or translating long documents (over 35,000 words).

Highlights

• High Efficiency: 1.9B parameters, yet competitive with larger models in various benchmarks.
• Long Context Support: The 32K variant can process documents of over 35,000 words in one go.
• Multilingual Strength: Strong translation abilities, particularly for East Asian languages, due to the inclusion of internet community corpus.
• Role-Playing Framework: Includes a dedicated framework for creating customized characters using RAG and dialogue corpora.
• Broad Compatibility: Adapted for use with transformers, llamacpp, and Ollama.