The Usability of Augmented Reality

Augmented reality (AR) enhances the real world with an additional layer of information. This novel technology has great potential and applications in various contexts and domains. However, designers and practitioners have limited guidelines and principles for successfully designing and implementing AR experiences.

We conducted a mobile usability-testing study with 4 remote and 7 in-person participants to get insights into issues and opportunities surrounding the user experience of mobile AR apps. For this study, we tested variety of AR applications from various domains such as fitness, entertainment, ecommerce, tourism, art and history, education, and science.

This article is the third in a 3-part series discussing the user experience of augmented-reality features in mobile apps. The first article talks about onboarding users in an AR app; the second article presents recommendations for guiding users through the process of calibration.

What Do Users Think About AR?

Most users in our study were unfamiliar with augmented reality; some confused it with virtual reality. They had little previous experience with AR and, as a result, little knowledge about AR-related patterns or processes (e.g., calibration). Some users who had previous gaming exposure were able to quickly figure out the AR apps that we tested; others struggled even with the most basic interactions.

Overall, while users were generally enthusiastic about AR and about its potential, they also questioned the gratuitous use of AR in some apps.

Gaming Experience Transfers to AR

While AR is a relatively new technology, some AR patterns and signifiers were inspired by game user interfaces and interactions. In our study, we found that participants who had exposure to game UIs found  AR patterns easier to figure out, compared to the participants who did not play any games.

For instance, one participant who highlighted gaming as one of his primary mobile activities had no problem interacting with the DionsAR app.  After launching the AR experience, he easily interpreted the icons in the app: “So, I see the dinosaur under the X [icon]; I tapped the knife and fork [icon] — I assume that this would be him eating. Yeah, it looks like that was the motion of him eating. The Z [icon] would be what I would assume him in his sleeping position. Yep. And that was the case. And then the sword [icon] would be him being an aggressor or attacking.”

Best AR Experiences Are Immersive, Interactive, and Educational

AR has great potential for creating immersive experiences that educate the public or entertain it by telling a story. Participants repeatedly shared their excitement about immersive educational AR stories.

A user of Kinfolk app thought the app would be helpful to teachers and educators: “I am learning about historical icons in the AR mode. This app is cool. […] I think a lot of educators or people that may work in a school may be interested in something like that because they can create an activity with this app and because it seems very engaging.”

A different participant, who was using ARLOOPA, an AR app with a diverse library of 3D models, was also excited about the potential of the technology: “I think it’s super cool. I think this could be used for so many things like working in healthcare. I think people like, especially kids, they would be so consumed by this because it literally looks like there’s a cat right in front of you. And then I’m just looking at the other things that are in here. There’re so many categories that you can unlock. I love that. I feel like it could be good for educational purposes. I think it’s awesome. “

A study participant using the Gettysburg AR app, a history and storytelling app about Gettysburg and the American Civil War, was enthusiastic about the immersive experience that the app created. He said, “I just think it’s awesome. I think it really, like, makes the experience so much cooler. Like I said, it makes you feel like you’re there, you get a little insight into like how things were just by looking at the clothing and all the visuals in there. It looks like there’s [sic] a lot of prompts, you can get as much information from this as you want. […] I think the fact that you can interact, but you’ve got like such cool visuals. Like, it really makes me feel like you’re there. And […] even […] the gunshots, like, that’s so neat to me.”

Another participant who was using the Civilization AR was excited about its educational value and about its interactivity: “I think it’s really cool. I love the historical aspect. I can see when it’s from, where it was discovered, what kind of material it is, the size. I love that you can adjust the size.”

The AR experience was at its best when users could easily see virtual objects in their environment and effortlessly interact with them. Seamless interaction with the AR model increased participants’ satisfaction with the app.

Several participants mentioned that they would like to keep using AR technology in the future, as they found it helpful and entertaining.

AR Should Not Be Gratuitous

AR is a cool, new feature that may generate interest among users, but it should not be forced on an application. The AR feature should bring value to the interaction. Even though users find good augmented reality delightful, creating a usable AR interaction is no easy feat. Many of the AR apps that we tested had big usability issues and users will often not bother to use a clunky AR feature, especially when it adds little or no value to their experience.

For example, one participant using the Google Arts & Culture mentioned that viewing famous artefacts and paintings in the AR mode was not effective. She commented: “I don’t understand it […] The painting is awesome, but the fact that it’s in my living room […] seems ridiculous. And I don’t know why they came up with this. […] I mean, unless you have, like, a super-creative mind and you’re out in the world and you want to, like, juxtapose Moses and the 10 commandments by a river or a lake or something that would mean something artistically. But I think the average person who’s looking at this doesn’t want Moses in their living room. They just want to look at the picture and maybe zoom in. […] It doesn’t zoom and fill my whole screen. I can’t see super-fine details. It just makes it bigger and bigger in my living room. So, I don’t know. I think it’s weird.”

In certain apps, particularly in ecommerce or advertising, the AR experience can allow users to envision the product in their environment. In such cases, details such as realism of the model, sense of scale, and accuracy are highly significant. Technical limitations that prevent you from providing a realistic view of the product can backfire and adversely impact users’ perception of the product. Thus, if you do not have the resources to provide a high-quality AR experience, you can skip it altogether!

How to Improve the User Experience of AR Feature

This section outlines some of the challenges that our study participants struggled with when they interacted with AR features and provides design recommendations for addressing them.

Ensure Discoverability and Findability of AR Features in Ecommerce Apps

Among the biggest issues identified by our previous ecommerce AR study were discoverability and findability. They both continue to be common challenges for users. Even though companies have invested time and resources in AR features, many of our study participants were unaware that some of their most frequently used apps (such as Amazon, Ulta, Best Buy, or Target) had such features. Once they used such features, almost all were excited and could see their potential but, since AR was not consistently used for all products in an ecommerce app, they had a hard time finding those items for which the AR features were available.

For example, one participant who was using the Houzz app tried to find a product with the AR feature. After a couple of minutes of search, she stated: “I’m just still looking […]. It’s hard to find one, I guess that I’m just looking to see if they have any […] the ones that I liked, they don’t have it.” She added: “Since a lot of items in the app did not give you the AR option, it was sometimes frustrating… I had to search out a whole lot of different items just to find the AR.”

To make it easy for users to locate those products with an AR feature, consider adding a special visual indicator to their listings on the product-listing page. Alternatively, allow users to narrow down listings to only those products that have an AR feature available.

Additionally, AR features were often signified by unfamiliar, cryptic icons. So, even if users did find a product with an AR feature, they could miss it because they did not recognize the corresponding icon.

Our study participant skipped a couple of AR-compatible products in the Etsy app because she did not recognize the icon used for AR. Eventually, she discovered it: “Let’s try this. This may be it. I don’t know what this symbol is, here. It’s a whole bunch of arrows.” 

An easy fix is to make sure that the AR icon has a good label that helps users interpret it.

Consider Users’ Context and Limitations

Before creating any AR experience, consider the context in which the app will be used and the constraints that may be imposed by that context. If the user must keep distance from the device, the app should not require them to read text instructions or touch the screen in order to interact with the UI.

Instead, it may be able to provide audio instructions and allow users to interact with the app through speech recognition or body gestures (waving, swiping, or raising their hands) instead of touching the screen. For instance, the Plaices-AR fitness game instructed the users to raise their hands to start the AR experience rather than touching the screen.

Consider also whether your app will accommodate all users. For example, wearing glasses might interfere with some filters or live-try-on features. Notify the users in advance about the requirements of the experience or consider alternative ways to assist them through the process.

Use Clear Instructions and Signifiers

Many users have little experience with AR and are unfamiliar with signifiers that someone who has reviewed several AR apps may consider standard. The user needs to be able to easily see the AR 3D objects, environment, and the prompts that guide her through the AR experience. This can be achieved through salient colors, combined with clear, unambiguous, readable instructions.

Another participant was initially pleased with the AR feature in the Best Buy application. However, later, he had trouble moving the AR object from one wall to an adjacent one in the environment since the cues and instructions were inadequate. He mentioned: “It’s annoying to turn the TV around. Navigating the TV around the corners […] is so annoying. It’s not good, actually.[…] [After figuring it out] Okay. So yeah. You don’t need to pinch it. Just automatically, it finds the right place. Yeah, I didn’t know it!”

The Mission to Mars application also provided understandable instructions combined with an effective and simple calibration process, which helped the user launch the AR experience. The user stated: “So they’re telling me the steps. […] There’s a couple of steps, so make sure the surface is good. Make sure the lighting is good and the camera is steady and tap the screen to place the object. Okay. I feel like I know what I’m doing.”

Ensure that Text and Controls Are Visible Across Different Backgrounds

In an AR scene, text, instructions, and UI controls are often projected on top of physical environment that is chosen by the user. These elements may not be legible or even visible if the user chooses a busy, low light background. It’s a good idea to add a solid background to these elements to ensure that they can be easily seen independent on the physical environment of any given user. 

Make the AR Feature Self-Sufficient

Even in the best-case scenario, when the AR experience is smooth and well-thought, placing a virtual object in a real environment comes with some cost to users, as they may need to find the right location for the object (and, in some instances, also go through the process of calibration). Thus, it’s a good idea to allow users to make some adjustments to the virtual object within the AR scene. In other words, the design of the AR interaction needs to be self-sufficient: the user should not have to quit the AR experience in order to find additional information (such as dimensions) about the object or to make small adjustments such as changing the color or other similar details.

The UI Should Not Interfere with the AR Feature

Even though UI controls and overlays are helpful because they allow users to customize the AR scene (as shown above), these options should not crowd the screen and compete with a seamless AR experience.

In an AR feature, minimize the amount of space covered with instructions, overlays, or UI controls. While all these elements have their role, they might also interrupt the primary task of the user — especially on a tiny mobile screen.

 For example, in the Amazon app, while the overlay hosted useful controls such as the Add to Cart or Details buttons, it also hindered the main task of viewing and interacting with the AR object in the real-life environment.

Consider Audio Narration to Increase Immersiveness

While audio is not for every app, there are several types of situations where having an audio option available could help create a better AR experience. First, like mentioned before, it can help users who may not be able to read instructions or other information displayed on the screen (e.g., because the device is placed too far away).

Second, if the app is striving to create an immersive experience (for instance, because it is telling a story or recreating an event), audio narration can be used to free up some valuable screen space and thus frontload the AR scene.

For example, a participant using the JigSpace app to learn about combine harvesters said, “So that was really cool. I think that maybe the only thing that could have been better is [if] I could have somebody read this to me. […] So that way I could kind of look at the jig itself and be told something, instead of having to, like, go down [towards the bottom of the screen to read the text] and then go back to the jig; but it’s not like a huge deal, but yeah. It’s cool.”

Another participant was excited about the audio narration in the Gettysburg app: “Wow! That’s so cool. I feel like the audio makes you feel like you’re there. Like you get kind of lost in it. And then I see some prompts where I can talk to her and learn more about Gettysburg. If I wanted to, or I can hear about this man’s last moments.”  

Most users are still unfamiliar with AR technology and, unless they have extensive gaming experience, they may be challenged by the icons and patterns used by AR features. If your mobile app uses AR, make users aware of this feature and help them easily locate those items in your app for which it is available.

Help people interact seamlessly with your virtual objects by using clear signifiers, text labels for icons, and step-by-step instructions that are easy to see. Allow them to easily modify the AR object within the AR scene, to prevent the need for recalibration, but ensure that UI icons leave enough room for the AR scene.

Last but not least, even though most users find delight in augmented reality, do not implement AR for the sake of AR, but, rather, make sure that it adds value to the overall user experience.