TL;DR:
- Large Language Models (LLMs) and Generative AI are gaining popularity, leading to a surge in open-source LLM projects.
- A recent study by Rezilion focused on these projects and identified significant security concerns.
- Trust boundary risks, data management risks, inherent model risks, and basic security best practices all pose potential threats.
- The average security score among assessed projects was only 4.6 out of 10.
- Projects gaining rapid popularity are more susceptible to security risks.
- Recommendations were provided to mitigate risks and enhance long-term safety.
Main AI News:
As the AI domain experiences a surge in the popularity of Large Language Models (LLMs) and Generative AI, businesses and individuals alike are eager to capitalize on this technological wave. However, amidst the excitement surrounding these advancements, it is crucial to focus on the security aspects, particularly when it comes to open-source LLMs. The rapid adoption of this technology across various use cases necessitates a closer examination of the associated risks.
In a recent study conducted by Rezilion, a renowned automated software supply chain security platform, experts have delved into this very issue, unearthing surprising findings. The research focused on projects that met specific criteria:
- Projects were created within the past eight months (approximately from November 2022 to June 2023, as of the publication of this paper).
- Projects related to LLMs, ChatGPT, OpenAI, GPT-3.5, or GPT-4.
- Projects with a minimum of 3,000 stars on GitHub.
These criteria ensured the inclusion of major projects within the research scope. To present their findings, the researchers employed the OpenSSF Scorecard, a comprehensive framework developed by the Open Source Security Foundation (OSSF). This tool aims to assess the security of open-source projects and facilitate improvements. The assessment encompasses various aspects of the repository, including vulnerabilities, maintenance frequency, presence of binary files, and more.
The objective of these checks is to ensure adherence to security best practices and industry standards. Each check is assigned a risk level, representing the estimated risk associated with non-compliance. The cumulative score obtained from the 18 checks is divided into three themes: holistic security practices, source code risk assessment, and build process risk assessment. The OpenSSF Scorecard assigns an ordinal score between 0 and 10, along with a corresponding risk level score for each check.
Remarkably, the study reveals that nearly all open-source LLMs and associated projects exhibit significant security concerns, which can be categorized as follows:
- Trust Boundary Risk: This category encompasses risks such as inadequate sandboxing, unauthorized code execution, SSRF vulnerabilities, insufficient access controls, and even prompt injections. These risks involve the concept of trust boundaries, whereby malicious actors can inject harmful NLP commands that propagate through multiple channels, severely impacting the entire software chain. A notable example is the CVE-2023-29374 vulnerability in LangChain, which is the third most popular open-source LLM.
- Data Management Risk: Data leakage and training data poisoning fall within the purview of data management risks. These risks extend beyond Large Language Models and pertain to any machine-learning system. Training data poisoning involves the intentional manipulation of an LLM’s training data or fine-tuning procedures by attackers to introduce vulnerabilities, backdoors, or biases that compromise the model’s security, effectiveness, or ethical behavior. This malicious act aims to undermine the integrity and reliability of the LLM by injecting misleading or harmful information during the training process.
- Inherent Model Risk: These security concerns arise from limitations inherent in the underlying ML model, including inadequate AI alignment and excessive reliance on LLM-generated content.
- Basic Security Best Practices: This category encompasses issues such as improper error handling and insufficient access controls, which align with general security best practices. These issues are not unique to LLMs specifically but are prevalent across various machine learning models.
Alarming yet enlightening, the research reveals the security scores obtained by these models. The average score among the assessed projects was a mere 4.6 out of 10, with an average age of 3.77 months and an average of 15,909 stars. Interestingly, projects that gain rapid popularity are at a significantly higher risk compared to those developed over an extended period.
In addition to identifying the current security issues, the company has provided extensive recommendations in its research to mitigate these risks and enhance long-term safety. By conducting comprehensive risk assessments and implementing robust security measures, organizations can leverage the potential of open-source LLMs while safeguarding sensitive information and maintaining a secure environment.
Conclusion:
The analysis of open-source LLM projects reveals alarming security risks that should not be ignored. The findings suggest that while there is a significant interest and enthusiasm for LLM technology in the market, there is a pressing need to prioritize security measures. Organizations must proactively address trust boundary risks, data management risks, inherent model risks, and basic security best practices. Failure to do so may result in compromised integrity, data leakage, and vulnerability to malicious attacks.
By adopting comprehensive risk assessments and robust security measures, businesses can strike a balance between harnessing the power of open-source LLMs and safeguarding sensitive information. This highlights the importance of ensuring proper administration of security protocols and adopting the suggested changes to mitigate risks effectively. Ultimately, businesses that prioritize security will be better positioned to navigate the evolving landscape of AI technology while maintaining a secure environment for their operations and stakeholders.
