How do hundreds of fish manage to swim together in remarkably coordinated schools without any leader? This question has fascinated biologists for decades and remains one of the major challenges in the study of collective animal behavior. Researchers from the Research Center on Animal Cognition (CRCA), the Laboratory of Theoretical Physics (LPT) and the Toulouse Institute of Computer Science Research (IRIT) have now developed an innovative approach combining mathematical modeling with interactive virtual reality to answer it. Their study, published in PLoS Computational Biology, shows that it is now possible to reconstruct the interaction rules governing collective swimming and validate them experimentally by allowing a real fish to interact in real time with its digital twin. This work was featured on the CNRS Institute of Biology website.
Fish schools can change direction almost instantaneously, with hundreds of individuals responding as if they were a single organism, despite the absence of any leader. Scientists have long known that this coordination emerges from local interactions between neighboring fish. However, identifying the precise behavioral rules that each fish follows—and demonstrating that these rules are correct—has remained an outstanding challenge.
To tackle this problem, the researchers combined animal experiments, mathematical modeling, and immersive virtual reality. Their goal was not simply to reproduce collective behavior in computer simulations, but to demonstrate that the interaction rules embedded in the mathematical model genuinely correspond to the mechanisms real fish use when coordinating their movements.
The team first recorded the three-dimensional trajectories of pairs of fish swimming freely in a hemispherical aquarium. By analyzing these trajectories, they reconstructed the social interactions governing their movements: long-range attraction, short-range repulsion, alignment with neighbors, and avoidance of the aquarium walls. Fish continuously adjust not only their direction and speed, but also their position in the water column according to both their neighbors and their surrounding environment.
These interaction rules were then incorporated into a mathematical model that accurately reproduces the observed three-dimensional behavior. Simulations faithfully captured swimming speeds, distances between individuals, body alignment, swimming depth, and even the spectacular collective U-turns that characterize schooling fish.
The most innovative aspect of the study, however, goes far beyond comparing simulations with experiments. The researchers developed a closed-loop virtual reality system that allows a live fish to interact in real time with one—or even several—virtual companions controlled entirely by the mathematical model.
A 3D camera continuously tracks the movements of the real fish. These data are immediately transmitted to the mathematical model, which computes the response of the virtual fish. The resulting image is projected onto the curved inner wall of the aquarium using a three-dimensional anamorphic projection, creating the illusion that a real fish is swimming alongside the animal.
Figure: A fish swims freely inside a hemispherical aquarium while its three-dimensional movements are continuously tracked in real time. These data are fed into a mathematical model that controls a virtual fish projected onto the aquarium’s inner wall. Using three-dimensional anamorphic projection, the image of the virtual fish (top) is continuously distorted according to the position and orientation of the real fish (bottom), so that it appears to be a genuine companion swimming in the same volume of water. The virtual fish constantly adjusts its position and orientation in response to the movements of its living partner, providing a powerful tool for investigating the mechanisms underlying coordinated collective swimming. Photo: David Villa, ScienceImage/CBI/CNRS, Toulouse.
Video: A real fish swims freely inside a hemispherical aquarium while interacting in real time with the virtual avatar of a conspecific. The avatar’s movements are computed by a mathematical model using the real fish’s instantaneous trajectory. The control interface (upper left) displays the trajectories of both the real fish (red) and its virtual partner (blue), together with the model parameters. A 3D camera (lower left) continuously tracks the fish’s movements. On the right, the anamorphically projected image of the virtual fish creates the illusion of a real companion swimming alongside the animal.
Remarkably, real fish spontaneously coordinate their movements with these virtual partners. They adjust their speed, direction, and position just as they would when swimming with another fish. This represents a major milestone in the study of collective behavior. For the first time, interaction rules reconstructed from observations are not only capable of reproducing the statistical properties of animal trajectories, but also generate behavior realistic enough to elicit natural social responses from living animals.
The researchers have since extended this approach to allow a single fish to interact simultaneously with as many as nine virtual companions. More broadly, the work opens a new avenue for studying collective behavior in animal groups. By enabling living animals to interact directly with virtual agents whose behavior is entirely defined by mathematical models, researchers now have an unprecedented tool for quantitatively testing competing hypotheses about social interactions.
Beyond fundamental biology, the approach could inspire the design of autonomous robots capable of integrating into animal groups, as well as bio-inspired collective systems such as swarms of drones or fleets of autonomous vehicles. More generally, it illustrates the emergence of a new generation of experiments in which living organisms and digital agents interact seamlessly, turning virtual reality into a powerful laboratory for uncovering the fundamental principles of collective intelligence.