CodeQwen1.5 Release Notes

TL;DR

Qwen has introduced CodeQwen1.5-7B, a specialized 7-billion parameter open-source code LLM built on the Qwen1.5 family. It supports 92 programming languages and provides a 64K token context window, demonstrating competitive performance against larger proprietary models in code generation, debugging, and SQL tasks.

Basic Code Generation Capabilities

CodeQwen1.5-7B narrows the gap between open-source models and proprietary systems like GPT-4 in basic code generation. In evaluations using HumanEval and MBPP, the model outperforms other 7B-scale open-source models such as CodeLlama-Base, StarCoder2, and DeepSeek-Coder-Base.

Benchmarking Performance

According to the provided data, CodeQwen1.5-7B (Base) achieved a 51.8 score on HumanEval 0-shot and a 72.2 score on MBPP 0-shot. The Chat version, CodeQwen1.5-Chat, reached 83.5 on HumanEval 0-shot and 77.7 on MBPP 0-shot, surpassing DeepSeek-Coder-Instruct (6.7B) in several categories.

Beyond Python, the model is proficient in eight mainstream languages evaluated via MultiPL-E: Python, C++, Java, PHP, TypeScript, C#, Bash, and JavaScript.

Long-Context Understanding and Generation

CodeQwen1.5 supports input lengths of up to 64K tokens, a capability developed by incorporating repository-level long sequence code data during pre-training. This allows the model to better understand repository-level code and function as a code agent.

Long-Context Evaluations

  • Perplexity (PPL): Testing on GitHub Trending repositories (as of 2024-03-28) showed that CodeQwen1.5 maintains a downward trend in perplexity as sequence length increases.
  • Needle in the Code: In a synthetic task where a custom function was inserted into the Megatron codebase, the model successfully replicated the function across a 64k length range.
  • SWE Bench: In this practical software development benchmark, CodeQwen1.5 achieved a score of 0.89, which the Qwen team notes surpasses ChatGPT-3.5.

Code Editing and Debugging

CodeQwen1.5 is designed to handle code modification tasks beyond simple generation. Using the CodeEditorBench suite, the model was evaluated across four dimensions: Debugging, Translation, Language Switching, and Code Polishing. The results indicate that CodeQwen1.5 achieves state-of-the-art (SOTA) performance for models at the 7 billion parameter scale.

Text-to-SQL Capabilities

CodeQwen1.5-Chat demonstrates high proficiency in translating natural language into SQL queries, positioning it as a tool for non-programmers to interact with databases.

Using the DIN-SQL prompting method on the Spider and Bird benchmarks, CodeQwen1.5-Chat ranked second, placing it close to GPT-4. The Qwen team attributes this performance to the use of scalable, verifiable, and varied synthetic data during both the pre-training and fine-tuning stages.

Technical Specifications

  • Model Size: 7 Billion parameters.
  • Pre-training Data: Approximately 3 trillion tokens of code-related data.
  • Language Support: 92 programming languages.
  • Context Window: 64K tokens.

Sources