Augmenting reality to make the unseen seen

When it comes to augmented reality (AR), more is not necessarily better. Especially if you’re a soldier using night-vision goggles while lugging heavy equipment through challenging environments. Defence scientists Larissa Cahill and Dr Shahd Al-Janabi are helping Defence, and particularly Army, explore operational concepts and understand the impact of augmented reality on our dismounted soldiers.

Cahill is approaching the topic from a research background in night vision and sensor-based systems over which augmented information is being placed. Al-Janabi is broadening that scope to include information from autonomous systems, and is generally more tech-agnostic.

“I’m looking at the general interface display and control aspects,” Al-Janabi explains. “There’s an image out there of the soldier of the future – receiving information from many different sources – that’s been created by console game developers and Marvel characters like Tony Stark in his Ironman suit. But the reality is that there’s a limit to how much information people can process at any one point in time. Our job is to use cognitive science to inform how AR displays are designed, and to test and evaluate different designs.”

Double-edged sword

Al-Janabi presented this research program at the 2018 Defence Human Sciences Symposium, generating interest and discussion around their paradigm of applying and testing existing visual processing knowledge in this new space. “There’s no point reinventing the wheel when we’ve already got a lot of theory we can leverage for system design,” says Cahill. “Well before AR we knew that people don’t process all the visual information they receive; we pick and choose and that leads to all sorts of design considerations. We are drawing on the significant research already out there in relation to land vehicle and aeronautical displays in particular, and understanding to what extent it is applicable to the dismounted soldier.”

According to Al-Janabi, augmented reality is a double-edged sword. “It has its formation in the gaming industry, when developers began to help players process information by presenting it in their line of sight. The design industry took this concept and decided it might be a good idea to present the users with all of the information, without a lot of consideration as to what information was needed, how it should be presented. Some designs may be far less advantageous than is assumed.”

While augmented reality design for seated gamers, pilots and vehicle operators seems to work well, soldiers face a whole different ballgame including movement after dark with night vision systems that are already visually exhausting. That’s where this duo bring their cognitive science skills into play. What advantage does a design provide and how can it be improved?

Iterative design process

This research is heavily influenced by technology availability, with augmented reality capability often oversold. “We use prototype systems on soldiers out in the field to explore information needs, while also undertaking fundamental technology agnostic work,” says Cahill. Design is a two-step process at the moment. The duo use virtual reality simulations in combination with Oculus Rift glasses, for example, as a representative head-mounted system. Results from the simulation inform physical prototype designs, which are then manufactured and tested by soldiers in real environments. And then it’s back to the lab for another iteration.

So what does the future hold for augmented reality systems? According to Cahill and Al-Janabi it comes down to what our appetite is for humans losing control. Everyone wants to make their lives easier by handing off the hard work and decisions to autonomous systems, but is that really the right thing to do? Do we want artificial intelligence algorithms filtering what soldiers see on their displays and guiding their decisions? Or making their decisions for them?

“The more we can do in this space to inform Army, the better placed they’ll be to make decisions about how to design, acquire and/or operate with this technology,” says Cahill. “If we don’t do some of this human factors research it will be left to the assumption that more is better.”