What is Fiori Android AR Annotation

Fiori Android AR Annotations components are designed for SAP mobile android applications to help users explore objects in a facility, warehouse, machinery, or a business object to enhance the user experience by leveraging augmented reality.

My Role

I am one of the designers for the Fiori Android AR annotation experience. This project is based on the design exploration from two other designers on the Fiori IOS AR annotation experience.

Challenge

(1) Based on the design exploration of the Fiori IOS AR annotation experience, we need to work with the developers to create the Android AR annotation experience based on the ARCore technology.
(2) Understand iOS ARCore's capabilities & limitations.

Product Related

Department: SAP mobile
Duration: 4 months
Designer: IOS research: Suhaib, Naomi ; Android: Mianying, Mina
User: SAP product designer / developer
Toolbox: Figma, After Effect, Android Studio

Final Design Preview

Background Acquisition

When our Android team took this project, we reached out to the AR IOS research team and understood why they picked AR annotation as the use case was because SAP already has an annotation AR demo developed by SAP IT based on some business use cases. Another reason is that their competitor analysis research shows that annotation is one of the common ways to apply AR technology in modern applications.

Design Constrain

Next, the IOS AR research team shared the design constraints with us based on the ARkit technology they explored and the challenge project is facing.
1. 3D assets not widely available unless core to the sales process.
2. Lack of clarity regarding who (OEM, customer) should develop experiences​.
3. Localization techniques still require visual anchors (images, object detection under some conditions) for stable, robust experience​.
4. Consumer-grade Mixed-Reality hardware not widely available​.

Design Opportunities

Beside the design constrain, they also share with us the opportunity they found in this project from the hardware and cooperation perspective.
1. Modern features of occlusion, persistence & collaboration provide a functional basis for advanced experiences.
2. All modern iOS devices support AR experience; the latest iPhone series includes LIDAR​.
3. ARKit is a focus technology for Apple, thus relevant for SAP in delivering best-of-breed apps.

Design Options

Based on the design options the IOS team concluded, they defined the design scope of the Fiori IOS AR-pattern as an annotation-centric mobile-based AR pattern. They explored the idea based on that foundation.

3D world space (Plane anchor)

Pro: Realistic, highly engaging, advanced rendering capabilities.
Cons: Only static content, hard to implement interactivity, significant jitter​​.

2D projected world space (Plane anchor)

Pro: Swift UI, very fast to develop, interactive, dynamic out of the box, still provides context.
Cons: Hard to read (jitter and movement), hard to place convincingly, no occlusion, won’t take scene properties.

2D Hybrid (Image anchor)

Pro: Balances discovering content in context with readability.
Cons: Complicated (but we do the work for you)​​.

Fiori IOS AR Components

The plane anchors only can place content vertically or horizontally. The​​accuracy and precision of persistence with plane anchors are highly variable. The IOS research team decided to use the 2D Hybrid design as the final design direction. Following their decision, they created the components for the IOS AR project, including the Initialization/Coaching component, Annotation markers/Transitional connector component, and Annotation card component.

Initialization / Coaching:
‍The coaching screen can help our users find the right image to start their AR experience. Users can expand the minimized instructions for the unobstructed view on demand.

Annotation card:
‍The annotation card contains the information related to the marker. If the user types the card, it will direct the user to the card detail page. But if the user types the marker, then the bi-directional interactivity helps in discovery, and the transitional connector directs attention to the right marker/card​.

Annotation markers / Transitional connector:
The annotation marker has select and unselect states.

ARcore Exploartion

Our team cooperate with the developer to go through the document and tech for the ARcore guideline. Compared with IOS that can place the AR elements in the 2D environment, all the elements in Android need to be put in the 3D environment, which means we need to consider z latitude (Depth) creating the design. I made a very simple prototype to explain if we want to make a 3D annotation similar to the IOS experience to help our developer know what it may look like.

Design Review

The card design under the research phase only provided a short description of the resources behind it. We want to give the designers more flexibility for the new card design to help their users better understand the information behind the card.

Card:
‍The card design under the research phase only provided a short description of the resources behind it. We want to give the designers more flexibility for the new card design to help their users better understand the information behind the card.

Annotation:
‍We want to redesign Annotation since the ARcore only places 3D elements in the AR environment. How to help AR annotation indicate the part more accurately and provide a better experience is the thing we want to verify.

In the end, our Android team decides to use the card with a preview image, title, and description to help users preview the content through the team's internal design review. We combine a small dot with the annotation to avoid the 3D jitter problem for the annotation.

Design Verification and Research

After we finalize our design, we start user testing with the research team to verify our design goal approach user's expectations, which is optimistic. Our next step would be the AR authorization flow to help the business user build their AR experience for their user.