Codestral Release Notes
Mistral AI has introduced Codestral, its first dedicated code model. Codestral is a 22B open-weight generative AI model designed to assist developers by generating code, completing functions, and writing tests through a shared instruction and completion API endpoint.
Broad Language Support and Core Capabilities
Codestral is fluent in over 80 programming languages. This includes widely used languages such as Python, Java, C, C++, JavaScript, and Bash, as well as more specialized languages like Swift and Fortran.
To improve developer productivity and reduce bugs, the model supports several key functionalities:
- Code Completion: Completing coding functions and partial code.
- Fill-in-the-Middle (FIM): A mechanism that allows the model to complete code based on the surrounding context.
- Test Generation: Automatically writing tests for existing code.
Performance and Benchmarks
As a 22B parameter model, Codestral is positioned to optimize the balance between performance and latency for code generation tasks.
Long-Range Context and Repository-Level Completion
Codestral features a 32k context window, which is significantly larger than the 4k, 8k, or 16k windows found in competing models. This larger window allows Codestral to outperform other models in RepoBench, a benchmark specifically designed for long-range repository-level code completion.
Technical Benchmarks
Mistral AI evaluated Codestral across several dimensions:
- Python: Performance was measured using HumanEval pass@1, MBPP sanitised pass@1 (generation ability), CruxEval (output prediction), and RepoBench EM (long-range completion).
- SQL: Performance was assessed using the Spider benchmark.
- Multi-language Support: The model was evaluated using HumanEval pass@1 across C++, bash, Java, PHP, Typescript, and C#.
- Fill-in-the-Middle: FIM performance in Python, JavaScript, and Java was compared against DeepSeek Coder 33B.
Deployment and Access Options
Codestral is available through multiple access paths depending on the use case:
Open-Weight and Licensing
Codestral is an open-weight model available for download on HuggingFace. It is released under the Mistral AI Non-Production License, allowing use for research and testing. Commercial licenses are available upon request from Mistral AI.
API Endpoints
- Dedicated Endpoint (
codestral.mistral.ai): Designed for IDE plugins and applications where users provide their own API keys. This endpoint is currently in a free beta period for 8 weeks and is gated behind a waitlist. - General API Endpoint (
api.mistral.ai): Available via la Plateforme, where queries are billed per token. This is intended for research, batch queries, or third-party applications that expose results directly to users.
Self-Deployment
Codestral is available for self-deployment through Mistral AI's sales team.
Ecosystem Integrations
Codestral has been integrated into several developer tools and frameworks:
- Application Frameworks: Integrated into LlamaIndex and LangChain for building agentic applications.
- IDE Integrations: Integrated into Continue.dev and Tabnine, allowing developers to use Codestral for chat and code generation within VSCode and JetBrains environments.
Community and Partner Feedback
Industry partners have highlighted the model's efficiency and accuracy. Notable feedback includes:
"Our internal evaluations show that Mistral's new Codestral model significantly reduces the latency of Cody autocomplete while maintaining the quality of the suggested code."
— Quinn Slack, CEO and co-founder of Sourcegraph
"We used Codestral to run a test on our Kotlin-HumanEval benchmark and were impressed with the results. For instance, in the case of the pass rate for T=0.2, Codestral achieved a score of 73.75, surpassing GPT-4-Turbo's score of 72.05 and GPT-3.5-Turbo's score of 54.66."
— Mikhail Evtikhiev, Researcher at JetBrains
Sources
- OriginalCodestral
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch