New Nonprofit Trillium Labs Aims to Conduct AI Research Transparently
Two industry scientists have launched a nonprofit called Trillium Labs that intends to study high-risk AI topics, including models improving themselves, while publishing results openly. The group argues transparency can help the wider community spot and mitigate risks that closed labs keep hidden.
Industry scientists Nathan Lambert and Tom Zick have founded Trillium Labs, a nonprofit that plans to tackle some of AI's more sensitive research areas while sharing its work publicly, breaking from the secrecy favored by many leading AI companies. The organization intends to examine topics such as recursive self-improvement, where AI systems help build better versions of themselves, along with the behavior of autonomous AI agents.
Unlike major AI developers, whose models are typically accessible only through apps or APIs with little disclosure about their inner workings, Trillium Labs says it will publish experiment details so outside researchers can study and reproduce them. Lambert argues that the secrecy common among frontier labs weakens the broader community's ability to scrutinize new techniques and limits outside experts' chances to help address emerging risks.
Early Focus on Post-Training and Reinforcement Learning
The nonprofit's initial work will center on post-training, the fine-tuning phase applied to large models after their core training is complete, with particular attention to reinforcement learning. That technique rewards models for desirable outputs and penalizes poor ones, making systems more capable but sometimes also more unpredictable or excessively agreeable. The lab will also study recursive self-improvement, a process that has drawn growing concern within the AI field over the possibility of humans eventually losing control of development.
Lambert and Zick met during the COVID-19 pandemic while both were graduate students researching AI at UC Berkeley. Lambert previously worked at the research lab Ai2 and at Hugging Face, while Zick worked at Harvard University and helped develop responsible-AI policy guidance at Charles Schwab.
Trillium Labs has already secured an undisclosed amount of funding from Schmidt Sciences, Halcyon Futures, and other backers. Its founders aim to raise between $40 million and $100 million overall and plan to put $30 million toward model training over the next 18 months.
