Researchers have trained a new kind of large language model (LLM) using GPUs dotted across the world and fed private as well as public data—a move that suggests that the dominant way of building artificial intelligence could be disrupted.
Flower AI and Vana, two startups pursuing unconventional approaches to building AI, worked together to create the new model, called Collective-1.
Flower created techniques that allow training to be spread across hundreds of computers connected over the internet. The company’s technology is already used by some firms to train AI models without needing to pool compute resources or data. Vana provided sources of data including private messages from X, Reddit, and Telegram.
Collective-1 is small by modern standards, with 7 billion parameters—values that combine to give the model its abilities—compared to hundreds of billions for today’s most advanced models, such as those that power programs like ChatGPT, Claude, and Gemini.
Nic Lane, a computer scientist at the University of Cambridge and cofounder of Flower AI, says that the distributed approach promises to scale far beyond the size of Collective-1. Lane adds that Flower AI is partway through training a model with 30 billion parameters using conventional data and plans to train another model with 100 billion parameters—close to the size offered by industry leaders—later this year. “It could really change the way everyone thinks about AI, so we’re chasing this pretty hard,” Lane says. He says the startup is also incorporating images and audio into training to create multimodal models.
Distributed model-building could also unsettle the power dynamics that have shaped the AI industry.
AI companies currently build their models by combining vast amounts of training data with huge quantities of compute concentrated inside data centers stuffed with advanced GPUs that are networked together using superfast fiber-optic cables. They also rely heavily on datasets created by scraping publicly accessible—although sometimes copyrighted—material, including websites and books.
The approach means that only the richest companies, and nations with access to large quantities of the most powerful chips, can feasibly develop the most powerful and valuable models. Even open source models, like Meta’s Llama and R1 from DeepSeek, are built by companies with access to large data centers. Distributed approaches could make it possible for smaller companies and universities to build advanced AI by pooling disparate resources together. Or it could allow countries that lack conventional infrastructure to network together several data centers to build a more powerful model.
Lane believes that the AI industry will increasingly look toward new methods that allow training to break out of individual data centers. The distributed approach “allows you to scale compute much more elegantly than the data center model,” he says.
Helen Toner, an expert on AI governance at the Center for Security and Emerging Technology, says Flower AI’s approach is “interesting and potentially very relevant” to AI competition and governance. “It will probably continue to struggle to keep up with the frontier but could be an interesting fast-follower approach,” Toner says.
Divide and Conquer
Distributed AI training involves rethinking the way calculations used to build powerful AI systems are divided up. Creating an LLM involves feeding huge amounts of text into a model that adjusts its parameters in order to produce useful responses to a prompt. Inside a data center the training process is divided up so that parts can be run on different GPUs and then periodically consolidated into a single, master model.
The new approach allows the work normally done inside a large data center to be performed on hardware that may be many miles away and connected over a relatively slow or variable internet connection.
Read the full article here
