Augmented reality technology helps non-speaking autistic population find their voice

For many autistic people, simply communicating with others can be a near-insurmountable challenge, with many not being able to speak at all.

A visionary project from the University of Calgary’s Schulich School of Engineering (SSE) aims to give this population a true voice.

This groundbreaking research is exploring the use of Microsoft’s augmented reality (AR) device called HoloLens 2, paired with a system a UCalgary team developed called HoloBoard, to train non-speaking autistic individuals with alternative communication modes by enhancing their motor skills.

It’s part of a project spearheaded by SSE researcher Dr. Diwakar Krishnamurthy, PhD, in collaboration with Dr. Mea Wang, PhD, of the Faculty of Science and Dr. Vikram K. Jaswal, PhD, of the University of Virginia, and in partnership with the non-speaking autistic community.

HoloLens 2 acts similarly to a standard virtual reality (VR) device with one main difference: the visor on the headset is transparent, allowing the user to see their real-world surroundings. Using the HoloBoard system, interactive holographic images emerge in the room via the headset. The concept of the system is for the user to practise their pointing and motor skills by interacting with the holographic keyboards and educational content that virtually appear in front of them.

Not only does the HoloBoard help users communicate, it can also help improve movement, explains Madison Imber, a Rapid Prompting Method (RPM) practitioner with Mentoring Minds, who has been working with autistic students for the past seven years. RPM practitioners like Imber customize the delivery of educational lessons to fit the unique sensory and motor profiles of a student — a technique that has informed the UCalgary team’s research.

“Many non-speaking autistic individuals also have apraxia, which, simply put, is a disconnect between the brain and motor movements,” Imber says. “Apraxia makes it difficult for many non-speakers to engage meaningfully in their environment without adequate supports. The HoloBoard system can provide such supports in a thoughtful way through holographic interactions. I have seen progress in many of the participants’ abilities to navigate 3D environments and interact with virtual objects.”

Working together for change

Collaboration between the researchers and subjects has been apparent throughout the process of developing the HoloBoard prototypes. Imber and her clients have been working with the UCalgary team since summer 2022 and input from the non-speaking autistic participants is deeply valued by the research team.

“In computer science and in software engineering, there's a term we use to describe this kind of research; we call it user-centred design, where we focus more on what the user actually wants,” says Dr. Lorans Alabood, PhD’22, a postdoctoral Fellow in the Department of Electrical and Software Engineering and lead researcher of the HoloBoard sub-project.

“My approach in designing the HoloBoard is aligned with this, but I have taken things a step further with an empathy-centred design. This is an approach which empathizes with the person using the system and requires us to understand our non-speaking users’ personal experiences in a deep way.”

The participants have been significant in the development of the prototype, and, in turn, they have demonstrated improvements in their motor skills and confidence.

“To see such progress and engagement with the HoloLens is truly amazing. I have also seen great progress in students’ belief and confidence in themselves to try new and exciting things,” says Imber.

Two trial users of the prototypes, Eric Herzog and Geoff Ondrich, using keyboards to communicate, say this research has already changed their lives. Ondrich points to individual letters on a page to comment, “My life is meaningful now.”

Herzog, meanwhile, uses a keyboard-to-speech tool to say, “I can now show the real me,” adding, “Thanks for believing in my voice.”

HoloBoard’s inspiring future

Herzog and Ondrich may have found the HoloBoard system to be useful in their social and educational pursuits, but the future of this technology holds countless benefits and opportunities for the non-speaking autistic population and beyond.

“This program is really important because the socioeconomic outcomes for this particular subpopulation is pretty bleak if the status quo in their support systems is continued,” says Krishnamurthy, also from the Department of Electrical and Software Engineering.

“So, if we can open up new opportunities for them through tech — it could be educational opportunities — downstream, the effects are going to be really positive, not only to these individuals, but also to their families and caregivers.” 

Alabood hopes classrooms will adapt holographic technology to better include autistic students.

“Currently, our classroom is designed in the very same way how it was 150 years ago,” says Alabood. “So perhaps more-inclusive setups where students can join a classroom holographically to accommodate their unique sensory-motor profiles should be considered from an institutional and community perspective.”