- Mozilla and EleutherAI hosted “The Dataset Convening” in Amsterdam on June 11.
- Aimed to promote openly licensed and open-access LLM training datasets.
- Emphasized the importance of open-access data as a public good.
- Highlighted the forthcoming “Common Pile” dataset by EleutherAI.
- Discussed legal compliance, ethical considerations, and best practices for open datasets.
- Addressed challenges in sourcing, curating, and governing open training data.
- Compared challenges in open datasets to early days of open-source software.
Main AI News:
In a groundbreaking move to address the challenges and opportunities within the open AI community, Mozilla and EleutherAI hosted an exclusive workshop, “The Dataset Convening,” on June 11 in Amsterdam. This event brought together 30 leading scholars and practitioners from open-source AI startups, nonprofit AI labs, and civil society organizations. The primary focus was on creating openly licensed and open-access LLM training datasets and overcoming the associated challenges.
Historically, the practice of sharing training datasets was common among AI developers. However, increasing competitive pressures and legal risks have made this practice almost obsolete. Mozilla and EleutherAI are championing a return to openness, drawing parallels to the transformative impact of open-source software on the internet. They believe that open-access data is a public good that can empower developers worldwide, fostering competition, innovation, and transparency.
The workshop saw participation from notable entities in the open LLM community, including developers of LLM training datasets such as Common Corpus, YouTube-Commons, Fine Web, Dolma, Aya, and Red Pajama. These datasets serve as blueprints for transparent and responsible AI progress, challenging the notion that performant LLMs cannot be trained without copyrighted material.
One of the highlights of the event was the discussion around “Common Pile,” a forthcoming dataset by EleutherAI composed solely of openly licensed and public domain data. This dataset builds on the success of its predecessor, “The Pile,” and aims to set new standards for openness and legal compliance in AI training data. EleutherAI also released a technical briefing and initiated a public consultation on Common Pile during the event.
The convening aimed to develop normative and technical recommendations for openly licensed and open-access datasets. Key discussion points included:
- Ensuring legal compliance and ethical outcomes while maintaining openness.
- Identifying best practices in sourcing, curating, governing, and releasing open training datasets.
- Addressing the challenges of sourcing public domain and openly licensed data, manual verification of metadata, and the legal status of data across jurisdictions.
- Exploring financial sustainability and infrastructural investments to support the development of open datasets.
Participants drew parallels between the current challenges faced by open datasets and those encountered in the early days of open-source software, such as data quality, standardization, and sustainability. The workshop emphasized the need for shared reference points and community collaboration to guide the development of open datasets.
In the coming weeks, Mozilla and EleutherAI will collaborate with participants to develop common artifacts and an accompanying paper to help researchers and practitioners navigate the complexities of advancing open-access and openly licensed datasets. These resources aim to strengthen the sense of community and support the ongoing efforts towards openness in AI.
The Dataset Convening is part of the Mozilla Convening Series, which brings together leading innovators in open-source AI to tackle critical issues and advance the community. The first event in this series, the Columbia Convening, focused on defining openness in AI. Mozilla remains committed to supporting communities invested in openness around AI and looks forward to fostering growth and collaboration in this movement.
With initiatives like these, Mozilla and EleutherAI are setting the stage for a more transparent, innovative, and collaborative future in AI development.
Conclusion:
Mozilla and EleutherAI’s initiative to advance open AI datasets signifies a pivotal shift towards transparency and collaboration in the AI sector. By promoting openly licensed and open-access LLM training datasets, they aim to foster innovation, competition, and ethical standards within the AI community. This effort not only addresses current challenges but also sets new standards for data accessibility and legal compliance, paralleling the evolution of open-source software in transforming technology development practices.
