Designing our Newest XR Experience
2019 - Present | Design Manager
My team designed the ThinkReality A3 experience from the ground up. We needed to have a design that was as elegant and sophisticated as the technology itself.
Our responsibility was to design the AR heads-up display launcher, mobile companion app, work applications, and a device management cloud portal. The final launcher is based on my design.
Design the user experience for the ThinkReality A3 that addresses critical issues from competitor products. Deliver the experience in 6 months for the first release.
Design a new AR experience (heads-up display, companion app, portal) that addresses critical user pain points. Rigorously test the designs with a diverse user base before delivering assets.
DESIGNING A NEW XR EXPERIENCE
"We were just asked to design the new AR experience for ThinkReality A3 and we only have 4 months to deliver it. Strap in, it's going to get bumpy. Let's have some fun!"
- Me to My Team in a Staff Meeting
- Also Me after said Staff Meeting
In December 2019, the AR/VR Business asked my UX team to design the Home Launcher experience. The ThinkReality A6 design was receiving negative customer feedback, especially around information architecture and visual hierarchy.
As the design manager, I had to first build a team that could support this request. I focused on identifying critical skills through a diverse lens. Creating inclusive technologies requires a diverse design team.
In 2020, I hired 8 additional resources to support this effort. Our multi-disciplinary design team included UX designers, interaction designers, an animator, developers, 3D prototypers, human factors engineers, and UX researchers that worked across 6 different countries and 8 time zones around the world.
We were truly a diverse team.
Before we jumped in, I told my team to take home a headset and test the current design. I asked them to take notes on their experience to see if our opinions matched the customer feedback.
Each member was asked to complete several tasks, ranging from first-time boot to repairing a PC using the PC Repair Application. Over the course of several weeks, we gathered these insights and detailed them out in our UX War Room.
THE CURRENT EXPERIENCE
We quickly discovered the current experience wasn't very usable and that there were several poorly designed features. This design was created in 2018 by an external design agency who lacked AR experience. Unable to rapidly prototype and test their designs, it had many issues that led to a poor user experience.
1. VISUAL CLUTTER
There are too many visuals on the screen at one time which leads to visual clutter. Some information was important while others were rarely used.
The overall design takes up too much of the field of view. This is a big issue with AR since too many visuals will block the user's real-world view.
2. INACTIVE vs. ACTIVE
The interface is filled with both active and inactive (read-only) buttons and icons that look similar. Although this treatment felt like visual consistency, it adversely affects the experience since many users cannot distinguish one set from the other.
3. NO VISUAL HIERARCHY
Since all the icons, buttons, and read-only information are the same size, there is no visual hierarchy, confusing users on what is important or where to look.
This treatment made tasks more difficult to accomplish since they are all competing for the user's attention.
4. AWKWARD CONTROLS
The basic controls are awkwardly designed. For example, the Volume and Brightness controls are steppers, which makes it difficult for users to quickly move from one end to the other.
Other controls such as toggles are vertically designed with color as the only feedback. For users with visual impairments like colorblindness, this makes it difficult to know the toggle state.
5. NO EYE REST AREA
The experience uses an eye gaze input method, which means that whatever object you look at, you select it. This also means that if you are reading an object, there's a risk you'll accidentally select it.
In this design, there are not many places to rest your gaze when searching for the right button or control. Without eye rest space, many users selected apps or switched toggles unintentionally.
INSIGHTS + FEEDBACK
We spent the next few weeks talking with users from around the world. We wanted to listen to their pain points first-hand and understand what tasks they were trying to perform. We learned a tremendous amount of information from these discussions that we hadn't heard before.
This is why it's critical for designers to talk directly with the end users. It's always better than receiving the information second-hand.
Next, we held multiple brainstorm sessions. I invited everyone to contribute ideas, regardless if they were a designer. This ensured everyone had a voice in the design process and gave space for them to be heard. And since we were extremely diverse, I knew everyone would come to the table with very different ideas. This inclusivity allowed everyone to participate in the creative process which is important in building team trust that's required in later stages of development.
After our brainstorm sessions, it was time to bring these ideas to life. Before we jumped into wireframes and mockups, we needed to understand the architecture of all the features and functions as well as the detailed workflows involved. We put together these initial architecture maps to help us stay lean and focused during the design process.
We also agreed to mockup the designs according to a set of defined user tasks (to establish a baseline) so that we can fairly compare designs against each other.
1. Home Launcher Experience
2. Change Brightness to Level 8
3. Connect to Lenovo Network
4. Launch Camera Application
5. Toggle Gesture Input to ON
6. Pin Camera Application to Home Menu
We also conducted a competitor landscape analysis to see the various interactions across the industry. We looked at all types of XR technologies to find patterns while conducting a proper comparion matrix of features and designs.
WIREFRAMES + MOCKUPS
The biggest challenge of creating these experiences is not fully knowing or understanding what the interactions would be like in 3D space. We referred to our user insights data, but many times we had to guess what the final 3D interaction would entail.
Below are various ideas created by me and my team.
CIRCULAR DESIGN CONCEPT
For this and many of my projects, I participated as both the design manager and as a designer. Based on the insights and data, I designed a circular interface and spaced out objects to minimize screen space.
I arrived at my concept because it solved two main issues:
1. It keeps the field of view open for critical information happening in the real-world
2. It provides an area to rest your eyes to avoid accidental selection with our gaze input modality
I focused on how users were going to use the device. Since it's a productivity tool, they were expected to perform important tasks in real-life. Directly blocking their FOV would affect their performance.
PHASE 1: EARLY PREFERENCE TESTING
It was time to get users involved. We spent several rounds doing internal experiment reviews (peer reviewing the protocols) before testing the concepts in user studies. We conducted a basic user preference test focused on:
1. Information Hierarchy (Architecture)
2. Intuitiveness & Cognitive Workload
4. Design & Aesthetics (User Interface)
Research methodologies included standard aesthetic testing (Aesthetics Pleasure of Design Scale), simple ranking and rating, and overall SUS score analysis with questions specifically targeting speed and accuracy.
The test materials were 2D graphical assets (mockups). We tested 3 unique designs and included the A6 Home Launcher design and a competitor interface as experiment controls. We took the constraints of 2D assets into consideration when analyzing the results since the fidelity added bias to the overall test. We acknowledged that looking at designs on a display is not reflective of the final 3D AR user experience.
"I think spacing out the elements and building layers of easy to understand controls will be essential. There are definitely some controls that are used more frequently than others."
"A lot of my hesitations in scoring some designs better than others is based on where my cursor is when I'm navigating. If I need to search for an object, I need a place to rest my eye without accidentially clicking something."
Based on the quick preference study, I was able to extract key insights to drive improvements to my circular design concept. I kept my wireframes neutral so we could focus on usability. Visual design treatments may inadvertently influence the user testing.
CIRCULAR DESIGN JUSTIFICATION
Here are a few reasons why I gravitated towards a circular design for this project:
A circular design would leave the center empty. This would avoid blocking the user's field of view and provides an eye-rest area to avoid accidental selection. This is important with unfamiliar interfaces. Users take longer to search for objects and need a place to rest their eyes.
This open area in the middle makes the design feel "light" since buttons are situated around the periphery. Visually, I avoided filling in the circles to preserve this airy effect.
In a circle, the distance between the cursor (defaults to center) and any object on screen is the same. In other designs, users may have to move their entire head to go from one side of the screen to the other.
After several rounds of internal design critiques and reviewing the data from the preference test, we chose 3 concepts to make lo-fi 3D prototypes. Our UX AR prototyper built the interfaces in 3D using a mix of Unity, Blender, and Adobe XD. We focused on activating the most critical paths for each task but also enabled most of the buttons to track error clicks during testing.
In addition to our own designs, we included three baseline prototypes to be used as experiment controls. The three additional designs were Daqri, Microsoft HoloLens, and our ThinkReality A6 Home Launcher.
Image Description: We created 6 lo-fi prototypes, including 2 competitor interfaces and the ThinkReality A6. Each concept was designed to complete each task so they could be fairly evaluated against each other. During testing, the concepts were randomized.
PHASE 2: FORMATIVE USER TESTING
Utilizing the lo-fi 3D Unity prototypes, we conducted our second round of user testing: this time, a more formalized study using internal stakeholders.
Our goal was to get user feedback on the prototypes, identify major themes and insights, and iterate our concepts before polishing them since each one took a lot of time to develop in Unity. We found several optimization techniques like using similar frameworks and assets to reduce the workload, but we were running short on time as the hardware schedule was being finalized.
One of our goals was to keep the test experiment and methodology consistent across phases so we could measure significant changes from one round to the next, particularly the usability and cognitive workload scores of our selected tasks.
The method we chose for this study was Think Aloud task-based testing that included time on task, accuracy, and task completion confidence. We used SUS-per-task scoring to measure usability and NASA TLX to measure cognitive workload as well as general ranking and rating scales.
Here are a few insights from the formal testing:
9 out of 10 in-person users for this study were developers or people very familiar with AR. We wanted to ensure our designs met the needs of our most experienced AR users. We made a note to include a more balanced experience participation pool in the next round.
Tasks were simply a subset of available functions and are not representative of all features in the entire experience
Certain interaction methods had cascading effects for subsequent task performance. We randomized the concepts, but some users may have seen the same concepts in the same order.
We were unable to control environmental conditions (e.g., lighting, body position, objects in real-world) since testing was done remotely. Differences in environmental conditions may have affected results.
"I really liked the free real estate in the center of the design. It prevents accidental selection, especially when you're unfamiliar with the layout like in this study. When I had to search for something in a task, I needed a spot to rest my cursor so I wouldn't accidentially trigger something."
Significant findings in the early user test allowed us to make critical changes to the designs. My circular design concept scored the best all-around, but as good researchers, we couldn't confidently confirm the results. After analyzing the data, we determined that the number of participants was too low, and the scores were not significant enough based on our pair-wise test. This was a difficult managerial decision because we were already behind schedule, but I knew we only had one chance to get this right, so I decided we needed an additional round of user testing with more participants.
Two of the concepts received significant redesigns taking into account the feedback from the study. The third concept, my circular design, which scored the highest, only needed a few updates.
I updated my designs to include user personalization, mini sub-controls, more colors with a focus on contrast and accessibility, and created visual hierarchy by differentiating objects based on their importance and frequency of use.
This version included read-only information in the center, a place for notifcations, and a quick trigger for voice input and feedback.
I also explored adding labels for accessibility and visual feedback on scales like Volume and Brightness.
PHASE 3: FORMATIVE USER TESTING
Using the updated 3D Unity prototypes, we conducted our third round of user testing. We worked hard on the experiment design to ensure that we could extract significant results and force a clear winner. We broke the study into three parts:
1. Internal Stakeholder Study via Head-Mounted Display (HMD)
2. External Online Global User Testing via PC
3. External Online Global User Survey
For the Internal Stakeholder Study, we recruited users who already had our A6 headset. The A6 was gaze-tech ready, which is the same techology the A3 uses. Participants were asked to download and install an .apk we developed for this study. The application was a self-assessed user study that walked participants through a series of tasks and had them answer questions about their experience. We also collected telemetry data such as where they looked, how long it took for them to complete a task, number of errors, and other data. The .apk was a true simulation of each of the concepts.
The method we chose for this study was Think Aloud task-based testing that included time on task, accuracy, and task completion confidence. We used SUS-per-task scoring to measure usability and Nasa TLX to measure cognitive workload and general ranking and rating scales.
For the second study, we used the same experimental design. Participants would walk through the same series of tests, but would do so on their PC. We created a tool that utilized the participant's camera to track head movement and linked this to the cursor to simulate the gaze cursor interaction. This was probably one of the coolest hacks we built to solve remote user testing.
Each of the prototypes allowed the user to select any object on the screen. For incorrect selections, an error appeared that allowed the user to go back to the previous menu. The user could end the task at any time by selecting the "Complete" button persistent on each screen. For this study, we also included two baseline prototypes: Daqri and the ThinkReality A6 interfaces.
Finally, for the third part of the study, we ran another online study that allowed participants to experience the prototypes without needing to install anything. We ran N=80 across various demographics, including users with/without experience in AR (differentiated between headsets and mobile). This was considered supplemental research to the previous studies because we couldn't control all the variables at play. By comparing sample sizes and extrapolating the data, we were able to decide if the results of this online study correlated to what we were seeing with in-person headset testing.