AR Stickers¶
There are situations when remote workers in the field need more information about a physical object or environment, such as a complex piece of machinery with many different physical components, for example. There may be relevant data such as descriptions, instructions, and/or video tutorials tied to each of these components. Some of these components may be replaced with new components in the future and require maintenance to keep the data updated. This AR Stickers module is designed to make it easier for a user to create, update, and visualize scenes with relevant information about a specific object or physical environment. AR Stickers supports two modes: edit-mode and view-only-mode.
Architecture¶
AR Stickers utilizes a handful of tools internally. It uses Android's ARCore and Sceneform libraries to provide the AR experience, androidx.navigation for navigating through fragments, and Retrofit library for network calls. These components are combined and maintained using the architecture design practices described by Android's guide to app architecture.
Getting Started with AR Stickers¶
Gradle Set Up¶
Add the following to your project-level Gradle file:
dependencies {
classpath 'com.google.ar.sceneform:plugin:1.17.1'
classpath 'androidx.navigation:navigation-safe-args-gradle-plugin:2.3.5'
}
Also, add the following to your app-level Gradle file:
dependencies {
implementation fileTree(dir: "libs", include: ["*.jar"])
implementation project(path: ':fiori-ar')
implementation project(path: ':fiori')
implementation 'com.sap.cloud.android:foundation:4.0.0'
def lifecycle_version = "2.2.0"
implementation "androidx.lifecycle:lifecycle-extensions:$lifecycle_version"
implementation 'androidx.lifecycle:lifecycle-viewmodel-ktx:2.3.1'
implementation "androidx.lifecycle:lifecycle-livedata-ktx:$lifecycle_version"
def navigation_version = "2.4.0"
implementation "androidx.navigation:navigation-fragment-ktx:$navigation_version"
implementation "androidx.navigation:navigation-ui-ktx:$navigation_version"
implementation "androidx.navigation:navigation-runtime-ktx:$navigation_version"
}
Create an Application to Initialize the ArSceneRepository¶
ArSceneRepository is an interface that plays as the single source of truth for the AR Stickers module. The client application may create an implementation of this interface or may use the default ArSceneRepositoryImpl provided by the SDK. The ArSceneRepository is platform-agnostic as it is not Android-specific.
We will be using the default ArSceneRepositoryImpl implementation in this example. We need an Application class to instantiate an instance of ArSceneRepositoryImpl.
The important method to note here is ArSceneServiceLocator.provideArSceneRepository(), as it takes
three parameters: the application context, a CoroutineScope to handle tasks outside the Fragment lifecycle, and an OAuth2Config to help with authentication. All parameters in OAuth2Config are type String.
In your Application file, add the following code to create an instance of the repository layer:
import android.app.Application
import android.graphics.Bitmap
import com.sap.cloud.mobile.fiori.ar.data.ArSceneRepositoryImpl
import com.sap.cloud.mobile.fiori.ar.util.ArSceneServiceLocator
import com.sap.cloud.mobile.fiori.ar.network.OAuth2Config
import com.sap.cloud.mobile.foundation.authentication.AppLifecycleCallbackHandler
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.SupervisorJob
class ClientApplication : Application() {
private var oAuth2Config : OAuth2Config = OAuth2Config(BASE_URL, appID, appVersion, clientId, redirectUrl, authUrl, tokenUrl, requestURL)
val arAuthoringDataRepo: ArSceneRepositoryImpl<Bitmap>
get() = ArSceneServiceLocator.provideArSceneRepository(this, CoroutineScope(SupervisorJob()), oAuth2Config)
override fun onCreate() {
super.onCreate()
registerActivityLifecycleCallbacks(AppLifecycleCallbackHandler.getInstance())
}
}
Create a MainActivity¶
Create an Activity to host your NavHostFragment:
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
}
}
Also create the associated activity_main.xml:
<?xml version="1.0" encoding="utf-8"?>
<layout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto">
<FrameLayout
android:layout_width="match_parent"
android:layout_height="match_parent">
<fragment
android:id="@+id/nav_host"
android:name="androidx.navigation.fragment.NavHostFragment"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:defaultNavHost="true"
app:navGraph="@navigation/ar_stickers_demo_navigation"/>
</FrameLayout>
</layout>
Create a Navigation Graph¶
Next we need to create a navigation graph. Here is an example called ar_stickers_demo_navigation.xml:
<?xml version="1.0" encoding="utf-8"?>
<navigation xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
app:startDestination="@id/client_fragment">
<fragment
android:id="@+id/client_fragment"
android:name="com.sap.cloud.mobile.fiori.demo.arstickers.ClientFragment"
tools:layout="@layout/fragment_ar_scene_list"/>
</navigation>
Now we are ready to implement edit-mode and/or view-only-mode.
Edit-Mode Example Implementation¶
The main entry point for edit-mode is EditImagesPagerFragment. Client applications are only required to navigate to this fragment to get started. The rest of the flow is handled internally. Use any Fragment in your current application as a starting point to take the user into an edit-mode.
Navigating to EditImagesPagerFragment¶
In ar_stickers_demo_navigation.xml add the following line below the fragment element. This line allows the current navigation graph to navigate to a sub-navigation graph called navigation_ar_stickers_edit_mode, provided by the SDK:
<include app:graph="@navigation/navigation_ar_stickers_edit_mode" />
Here we have a Fragment named ClientFragment that uses a Button to navigate the user to the EditImagesPagerFragment. The button instantiates a navigation Action named actionClientFragmentToEditMode (which is already defined in ar_stickers_demo_navigation.xml). We also have to let the repository layer know that we have selected a scene by calling selectScene(Int, Bool), which takes an integer parameter as the scene ID and a Boolean indicating whether we are going to edit-mode. Note that the SDK does not know about any existing scenes on the server. It is up to the client application to provide these scene IDs (as integers) or pass -1 as the argument to create a new scene:
import android.os.Bundle
import android.view.*
import androidx.appcompat.widget.AppCompatButton
import androidx.fragment.app.Fragment
import androidx.navigation.fragment.NavHostFragment
class ClientFragment : Fragment() {
val viewModel = ...
override fun onCreateView(
inflater: LayoutInflater,
container: ViewGroup?,
savedInstanceState: Bundle?
): View {
val view : View = inflater.inflate(R.layout.fragment_client, null)
val button = view.findViewById<AppCompatButton>(R.id.button)
button.text = "Create or Edit"
button.setOnClickListener {
// Pass in a known scene Id here to edit a scene, or pass -1 to create a new scene.
navigateToCreateOrEditArScene(-1)
}
return view
}
private fun navigateToCreateOrEditArScene(arSceneId: Int) {
val direction = ClientFragmentDirections.actionClientFragmentToEditMode()
// Make sure to let the repository layer know about the selected scene, as well
// as passing a Boolean value in the second parameter (true if editing, false if view-only).
viewModel.repository.selectScene(arScene.id, true)
NavHostFragment.findNavController(this).navigate(direction)
}
}
Also create the associated XML:
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:layout_width="match_parent"
android:layout_height="match_parent">
<androidx.appcompat.widget.AppCompatButton
android:id="@+id/button"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintTop_toTopOf="parent"/>
</androidx.constraintlayout.widget.ConstraintLayout>
View-Only-Mode Example Implementation¶
The main entry point for view-only-mode is ScanGuideFragment. Client applications are only required to navigate to this fragment to get started. The rest of the flow is handled internally. Use any Fragment in your current application as a starting point to take the user into view-only-mode.
Navigating to ScanGuideFragment¶
In ar_stickers_demo_navigation.xml, add the following line below the fragment element. This line allows the current navigation graph to navigate to a sub-navigation graph called navigation_ar_stickers_viewonly_mode.xml, provided by the SDK:
<include app:graph="@navigation/navigation_ar_stickers_viewonly_mode" />
As in the previous example, we can use the same ClientFragment with minor changes (the XML file remains the same as that of the previous example):
import android.os.Bundle
import android.view.*
import androidx.appcompat.widget.AppCompatButton
import androidx.fragment.app.Fragment
import androidx.navigation.fragment.NavHostFragment
class ClientFragment : Fragment() {
val viewModel = ...
// Example scene id.
var arSceneId = 3820239043
override fun onCreateView(
inflater: LayoutInflater,
container: ViewGroup?,
savedInstanceState: Bundle?
): View {
val view : View = inflater.inflate(R.layout.fragment_client, null)
val button = view.findViewById<AppCompatButton>(R.id.button)
button.text = "View Scene"
button.setOnClickListener {
// Pass in your known scene Id to view a scene.
navigateToViewArScene(arSceneId)
}
return view
}
private fun navigateToViewArScene(arSceneId: Int) {
val direction = ClientFragmentDirections.actionClientFragmentToViewonlyMode()
// Make sure to let the repository layer know about the selected scene, as well
// as passing a Boolean value in the second parameter (true if editing, false if view-only).
viewModel.repository.selectScene(arScene.id, false)
NavHostFragment.findNavController(this).navigate(direction)
}
}
End User Experience: How to Create or Edit an AR Scene¶
The following is a breakdown of the edit-mode user experience for creating, editing, and publishing an AR scene:
- Capture or select an image to represent the Reference Anchor
- Create AR cards
- Enter into AR mode and scan the Reference Image
- Add and place anchors near the scanned Reference Image
- Preview the anchors
- Adjust anchors if needed
- Publish/upload
1. Capture or Select an Image to Represent the Reference Anchor¶
Every AR scene needs exactly one mandatory anchor to be used as a "reference" for all other anchors, as it is treated as the origin (x=0, y=0, z=0) for these anchors to be placed relative to. All the user needs to do in this first step is click on the Capture button to select or capture an image. It is recommended to use an image with lots of distinct recognizable features such as a sticker. This image will be scanned when the user enters AR mode (step 3).
2. Create AR Cards¶
Each anchor in AR (besides the reference anchor) will be associated with an Annotation Card that displays additional data about the physical object that anchor is tied to. Data includes a title, description, URL, language, action text, and a preview image. The user will need to provide this information upon creation.
3. Enter Into AR Mode and Scan the Reference Image¶
After the user has completed creating the Reference Image and the Annotation Cards, they are then able to enter AR mode to create a scene with that new data. The first thing they will be prompted to do is scan the newly created reference image with their camera (this is all done while in AR mode). A green checkbox should appear when the image is successfully tracked.
4. Add and Place Anchors Near the Scanned Reference Image¶
Now that the Reference Image is being tracked in AR, it is time for the user to add anchors by clicking on the + button at the top end of the screen. Clicking this button will show a BottomSheetDialog containing a CollectionView of the recently created Annotation Cards. The user is able to select an Annotation Card they wish to place in the AR Scene and a blue sphere should appear in the center of the screen. The user is able to move their device around to move the sphere and "drop" it when they feel it is in an appropriate position (they are also able to drag it once it is dropped). They can repeat this process until all Annotation Cards are placed as anchors.
5. Preview¶
Clicking the Preview button will convert all the spheres into AnnotationMarkers, showing RecyclerView of Annotation Cards at the bottom of the screen. This is what the end-user (such as a field worker) in view-only-mode will also see and interact with. The user can exit preview mode if they wish to continue editing.
6. Adjust Anchors if Needed¶
Clicking the Back button allows the user to adjust the anchor by dragging it with one finger.
7. Publish¶
Once the user is satisfied with the scene, they are able to publish it to the server.
Tips¶
For best results, try to keep child anchors no further than a few feet apart for best results. It is not required, but it is recommended, to try to keep the reference image in the camera's view. This ensures more accurate placing with all anchors.
Further Reading¶
Here are additional sources on the ARCore, AugmentedImage class, Sceneform, Android Navigation Component, and architecture design, respectively: