TL;DR:
- Stability AI unveils Stable-Code-3B, a 3 billion parameter Large Language Model (LLM).
- Designed for code completion in 18 programming languages with a 60% smaller footprint than CodeLLaMA 7B.
- Features Fill in Middle Capability (FIM), supports 16,384-token sequences, and utilizes advanced technology like flash-attention.
- Trained on 1.3 trillion tokens, optimized with AdamW in bfloat16 precision, and operates with ZeRO-1 2D parallelism.
- Achieves 30% accuracy in key languages (CPP, Rust, Python, Java, PHP, and Javascript) surpassing other models.
Main AI News:
In a significant leap forward, Stability AI has introduced its latest innovation, the Stable-Code-3B model. This groundbreaking Large Language Model (LLM) is meticulously engineered to excel in code completion across a multitude of programming languages, all while offering a host of supplementary functionalities. Building upon the foundation laid by the Stable Code Alpha 3B, this remarkable model has been trained on a staggering 1.3 trillion tokens, encompassing both natural language data and extensive code repositories spanning 18 programming languages.
Notably, Stable-Code-3B distinguishes itself by delivering outstanding performance in a more compact form factor. When compared to its predecessor, the CodeLLaMA 7B, it shines as a 60% smaller yet equally potent solution.
At the core of Stable-Code-3B lies its auto-regressive language model, employing the revered transformer decoder architecture. But what truly sets this model apart are its advanced attributes, prominently featuring the Fill in Middle Capability (FIM). In addition, the model is engineered to handle sequences of up to 16,384 tokens, thereby accommodating extensive contextual information. Key enhancements include rotary position embeddings and a specialized tokenizer tailored for in-middle capability, complemented by various other tokens.
The comprehensive training regimen involved the utilization of vast open-source datasets, underpinned by a robust infrastructure featuring 256 NVIDIA A100 40GB GPUs. Optimized through the utilization of AdamW in bfloat16 precision, Stable-Code-3B seamlessly operates within a 2D parallelism framework with ZeRO-1, integrating cutting-edge technologies such as flash-attention and Rotary Embedding kernels from FlashAttention-2.
Through rigorous experimentation across a spectrum of programming languages, Stable-Code-3B consistently demonstrates remarkable efficiency, boasting an impressive 30% accuracy rate across CPP, Rust, Python, Java, PHP, and Javascript. While alternative models may marginally outperform it in specific languages or with larger-scale architectures, Stable-Code-3B remains unparalleled in its versatility and streamlined performance.
Conclusion:
Stability AI’s Stable-Code-3B represents a significant leap forward in the code assistance market. Its impressive 3 billion parameter model offers precision code completion in multiple languages while being remarkably compact compared to competitors. The incorporation of cutting-edge technologies and extensive training on diverse datasets enables Stable-Code-3B to outperform rival models in key programming languages. This innovation is poised to redefine the landscape of code assistance tools and cater to the evolving needs of developers and programmers across industries.
