Environmental Perception in a Swarm of Conversational Agents
We use LLMs to achieve collective perception of a pixelated environment in a swarm.
Collective perception, the process whereby multiple agents integrate partial local observations into a shared understanding of their environment, remains a core challenge in swarm robotics. Many existing approaches rely on numeric or categorical message passing, which effectively supports distributed consensus but limits the formation of rich, semantically grounded representations of the world. This paper addresses this limitation by introducing a framework for language-based collective perception using a swarm of agents powered by Large Language Models (LLMs). Each agent has limited local visual perception, and perform conversion of image input into linguistic representation. In addition, agents use local communication to share knowledge, resulting in improved individual understanding of the environment. The approach is evaluated by comparing each agent's knowledge over time to the ground truth built using global information. Using both individual and social learning achieves higher coverage of the environment on average than a swarm using only individual learning, demonstrating that natural language can serve as an effective coordination medium for decentralized perception. This work establishes a foundation for emergent, language-mediated world modeling in multi-agent systems.