TL;DR:
- Technology Innovation Institute (TII) has trained its open-source Falcon 40B model on Amazon Web Services (AWS).
- Falcon 40B is a 40-billion-parameter large language model (LLM) ranked #1 on Hugging Face’s Open LLM Leaderboard.
- The model was trained on 1 trillion tokens using Amazon SageMaker, a fully managed ML service.
- Customers can now access Falcon 40B through Amazon SageMaker JumpStart, eliminating the need to build their own model.
- Falcon 40B’s release as open-source empowers organizations to leverage its exceptional capabilities.
- The model’s architecture is optimized for inference, incorporating FlashAttention and multi-query techniques.
Main AI News:
In a significant stride towards AI innovation, Technology Innovation Institute (TII) has successfully trained its open-source Falcon 40B model on Amazon Web Services (AWS). Falcon 40B, a colossal 40-billion-parameter large language model (LLM), has garnered accolades as the top-ranked open-source language model on Hugging Face’s Open LLM Leaderboard.
The training process involved harnessing the power of Amazon SageMaker, a fully managed service that facilitates the development, training, tuning, and hosting of machine learning models, including LLMs. With the successful completion of this endeavor, TII’s efforts align perfectly with the UAE National AI Strategy 2031, demonstrating the country’s dedication to fostering AI innovation and making valuable scientific contributions.
Deploying Falcon 40B has now become effortlessly accessible for customers through Amazon SageMaker JumpStart, an extraordinary machine learning (ML) hub that offers a repository of pre-trained models. By leveraging Falcon 40B, users can harness its state-of-the-art accuracy and industry-leading performance, eliminating the need to create a model from scratch.
Dr. Ebtesam Almazrouei, Executive Director and Acting Chief AI Researcher of the AI Cross Centre Unit and Project Lead for LLM Projects at TII, expressed her excitement in a recent blog post, officially announcing the open-source release of Falcon-40B. She proudly stated, “We are delighted to unveil Falcon-40B, the world’s preeminent open-source language model.”
Wojciech Bajda, Managing Director for Public Sector Middle East and Africa at AWS, expressed his pride in collaborating with Technology Innovation Institute on the development of the Falcon LLM 40B model. Bajda highlighted the utilization of Amazon SageMaker to train this exceptional model. He further emphasized that the release of Falcon-40B as an open-source resource empowers organizations to leverage its extraordinary capabilities and drive advancements in AI-driven solutions, opening doors to new opportunities for progress.
Boasting an impressive 40 billion parameters, Falcon 40B stands out as an exceptional open-source model. Specifically designed as a causal decoder-only model, it underwent training on an extensive dataset comprising 1,000 billion tokens, including RefinedWeb enhanced with curated corpora. Falcon-40B is made available under the Apache 2.0 license, ensuring its accessibility and usability for developers and researchers alike. The architecture of Falcon-40B is meticulously optimized for inference, incorporating cutting-edge techniques such as FlashAttention and multi-query mechanisms.
Conclusion:
The successful training of the Falcon 40B model by Technology Innovation Institute on AWS, using the power of Amazon SageMaker, marks a significant milestone for AI innovation. The availability of Falcon 40B through Amazon SageMaker JumpStart provides customers with a pre-trained state-of-the-art language model, reducing the barrier to entry for AI applications. This development fosters advancements in the market by enabling organizations to leverage Falcon 40B’s exceptional capabilities, fueling progress and creating new opportunities for AI-driven solutions across various industries.
