The boundary between reality and the digital interface is becoming increasingly thin. The joint efforts of American scientists and Meta engineers have led to the creation of a technology capable of predicting exactly where a person will look in three-dimensional space, even before they turn their head. The new algorithm works in advance, forecasting the trajectory of attention several seconds ahead.
RBC-Ukraine reports this, citing materials from a scientific paper presented at the CVPR computer vision conference in Denver. The development radically changes the approach to creating interfaces for wearable electronics.
From Flat Images to Living Space
The key innovation lies in abandoning the analysis of static two-dimensional images in favor of full-scale modeling of human behavior in a real environment. The author of the study is Fiona Ryan, a graduate student at the School of Interactive Computing at Georgia Tech. She created the first 3D platform for predicting "scanpaths" — eye movement trajectories — based on first-person video.
"Since humans live in a three-dimensional world and are constantly in motion, standard 2D image analysis metrics cannot work effectively in a portable device like smart glasses," the scientist explains the essence of the problem.
The new algorithm views the attention vector as a sequence of gaze fixations that depend directly on the user's current goal. The system analyzes the context of actions: if it detects that a hand is reaching for a cup of coffee, it automatically calculates the operator's next step — searching for a place to put that cup.
Training on "Digital Twins"
The researcher performed the bulk of the practical work during an internship at Meta. To train the artificial intelligence, a specialized dataset called Aria Digital Twin was used. This dataset contains thousands of hours of first-person video recordings capturing everyday human interaction with objects within an apartment. The video is combined with high-precision 3D reconstruction of the entire room.
Thanks to this approach, developers were able to obtain the ideal coordinates of the actual gaze direction and match them with the geometry of the space, creating a reference model for training the neural network.
Eliminating Latency and Perspectives
At the current stage, the software is capable of stably predicting the direction of gaze an average of 3 seconds into the future. In certain simple scenarios, this figure reaches 10 seconds. This time interval is sufficient for the AR glasses' graphics processor to proactively generate (render) the necessary virtual hints or interface elements precisely in the zone where the person is about to look.
"This completely eliminates the image lag effect," notes Fiona Ryan, highlighting the main advantage of the technology for user comfort.
In the future, developers plan to integrate deeper contextual scenarios into the model — understanding exactly what the person is doing at the moment. This will allow for narrowing down prediction options during prolonged concentration on a single object.
The potential of the technology goes far beyond consumer electronics. In robotics, the algorithm can be used to train machines, allowing robots to copy natural human perception when performing household or industrial tasks.