Algorithmic advances now make it possible to use inexpensive cameras and motion sensors to simultaneously track a device’s location and map its surroundings.¹⁸ Advanced computer vision can also convert 2D images captured from moving cameras into detailed 3D models of spaces and objects¹⁹,²⁰ and use them to reconstruct photorealistic 3D scenes from any point of view,²¹,²² enabling the virtualisation of entire environments. AI can also layer these 3D models with semantic labels that categorise objects and features within the environment,²³ and seamlessly integrate virtual elements into scenes by allowing them to collide with or be occluded by real-world objects.²⁴ Many of these capabilities are now integrated into freely available software development kits from large technology and gaming companies.²⁵,²⁶

However, developing a truly immersive XR experience will require AI models that can go beyond simply mapping and reconstructing 3D environments to understanding and reasoning about them. Fortunately, solving these problems will be crucial for a wide range of AI applications, including self-driving cars, robots and drones, so considerable resources are already going into this problem. Vision-language models trained on large amounts of image and text data appear capable of some degree of spatial reasoning.²⁷,²⁸ But “world models”, which learn rich representations of an environment’s spatial and physical properties to make predictions of how it will evolve over time, could be even more promising.²⁹,³⁰

Many XR headsets today come with handheld controllers, but interaction is becoming increasingly multimodal. On many devices, camera-based gesture recognition and eye-tracking allow users to interact with virtual elements through simple hand movements or by using their visual focus as a cursor. Beyond providing more intuitive control options, the wide range of sensors embedded in leading XR devices can provide powerful insights into the user’s attention, intent and actions.

Machine-learning models can analyse eye-tracking data and ego-centric video captured by XR devices to infer what activity a user is engaged in³³,³⁴ or even to infer their intention to interact with both virtual and physical objects.³⁵ Eye-tracking data can also provide a window into the user’s attention and level of engagement.³⁶ When combined with basic physiological monitoring, it can also help to monitor the user’s cognitive load while carrying out tasks in XR.³⁷

By combining these user-centric insights with spatial information, virtual elements and control interfaces can be adapted to the user’s immediate context in real time.³⁸ This can be used to both improve the usability of virtual interfaces³⁹ and overlay helpful information and control options onto real-world objects.⁴⁰,⁴¹ It can even reduce the need for real-world physical interfaces, given the advances in “internet of things” connectivity between XR devices and the physical world. It can also make it possible for XR systems to detect when the user’s ability to interact with specific controls and interfaces is impaired by things like poor lighting, noise or multi-tasking, and adapt accordingly.⁴²

The rapidly improving reasoning and decision-making capabilities of large language models now allow them to take control of simulated bodies in virtual environments to solve a variety of tasks.⁴³,⁴⁴ The ability of these embodied “agents” to communicate in natural language also enables them to collaborate with each other, as well as with humans, to tackle challenges.⁴⁵ This is opening up the prospect of populating virtual environments with interactive AI characters for both entertainment and to act as digital co-workers.⁴⁶ Integrating agents into XR devices and providing them with access to their suite of sensors could also enable always-on digital assistants that see the world through the user’s eyes.⁴⁷

Considerable challenges remain. Even today’s most powerful models exhibit only limited embodied intelligence,⁴⁸ and our understanding of how AI and humans interact remains rudimentary.⁴⁹ Evidence suggests that creating useful digital assistants is challenging, and poorly designed AI can often hinder rather than help.⁵⁰,⁵¹

Nonetheless, the possibility raises a host of practical and ethical questions. Should AI agents be embodied in human form, and how will their appearance and personality reinforce social biases? Will users control their own agents, or will they be services provided by companies? Should agents be able to augment how others perceive the user in virtual spaces, adjusting their appearance, movements and even speech? And if agents can digitally alter the user’s XR environment to boost immersion or enhance productivity, how do we ensure this doesn’t lead to deception or manipulation?⁵²

A wide range of wearable haptic devices has been developed to simulate various tactile sensations such as pressure, texture and temperature.⁵⁴ These include gloves, vests and bracelets that rely on a variety of approaches to generate tactile illusions such as vibrating electromechanical devices, electrical stimulation or shape-changing materials. These devices can be cumbersome, though, which is spurring growing interest in “mid-air haptics”, which uses ultrasound beams to stimulate the skin and generate highly modifiable tactile sensations without physical contact.⁵⁵,⁵⁶ Another emerging approach involves using mobile robots to simulate virtual objects⁵⁷ or even reconfiguring physical environments to match the virtual one.⁵⁸

Simulating the chemical senses of taste and smell is now becoming feasible. So-called gustatory interfaces are making it possible to recreate taste sensations in virtual experiences.⁵⁹,⁶⁰ Olfactory interfaces can be combined with software that is able to recreate how smells evolve over space and time.⁶¹,⁶²

In the near term, haptic technology could play an important role enhancing the realism of mixed-reality training in fields like medicine.⁶³ But in the future it could also make XR experiences more immersive and boost their emotional depth.⁶⁴,⁶⁵,⁶⁶ However, this will require us to move away from using isolated haptic stimuli as an accessory to audio-visual experiences and instead work out how sequences of sensations can be combined to compose multisensory narratives.