Skip to content

Machine Learning

The SAPML framework includes components for developing machine learning features in apps, including drop-in UI components for text recognition, topology APIs for easy searching and filtering of text observations, and Core ML model management APIs for distributing custom Core ML models hosted on the SAP Mobile Services to the app. Apple Vision framework is used for text detection.

Text Recognition

The SAPML framework includes drop-in UI components for capturing text and displaying a preview of the text observations. FUITextRecognitionView is a subclass of UIView, which can be embedded anywhere in the view hierarchy, providing flexibility to control the frame of the view. FUITextRecognitionViewController is a convenience controller that embeds the FUITextRecognitionView in its entirety. Use this controller if it is not necessary to customize the frame of the FUITextRecognitionView.

Initialization and Configuration

Instantiate FUITextRecognitionViewController by using the default constructor. FUITextRecognitionView embedded in the controller can then be accessed as described below. Finally, present the controller.

let textRecController = FUITextRecognitionViewController()
let textRecView = textRecController.recognitionView
...
textRecController.onClose = {
    self.dismiss(animated: true)
}
present(UINavigationController(rootViewController: textRecController), animated: true)

Customize the appearance of the FUITextRecognitionView by changing the captureMaskPath or changing the properties of the OverlayView.

recognitionView.captureMaskPath = {
    let path = UIBezierPath(roundedRect: CGRect(x: 0, y: 0, width: 350, height: 220), cornerRadius: 4)
    path.lineWidth = 1
    return path
}()

recognitionView.overlayView.strokeColor = UIColor.white.cgColor
recognitionView.overlayView.backgroundColor = UIColor.black.withAlphaComponent(0.20).cgColor

Implementing observationHandler

FUITextRecognitionView is a general-purpose text recognizer that returns observations by calling observationHandler. These observations consist of all the texts that were recognized within captureMaskPath. However, the desired texts are only a portion of the observations. The filtering code can be written in observationHandler.

The following example uses NSDataDetector to detect the phone number of the observations. However, NSRegularExpression can also be used to run custom regex.

let detector = try! NSDataDetector.init(types: NSTextCheckingResult.CheckingType.phoneNumber.rawValue)
recognitionView.observationHandler = { [weak self] observations in

    let matches = detector?.matches(in: observations) ?? nil

    recognitionView.showTexts(for: observations, with: matches)

    if matches != nil {
        for match in matches! where match.resultType == .phoneNumber && match.phoneNumber != nil {
            DispatchQueue.main.asyncAfter(deadline: .now() + 0.8) {
                self?.textField.value = match.phoneNumber!
                self?.dismiss(animated: true)
            }
            return true
        }
    }

    return false
}

Alternatively, SAPMLTextObservationTopology can be used to get adjacent observations. Refer to Ordering and Filtering Observations for more information.

Using a Custom MLModel

Custom VNCoreMLRequest will be performed on the captured video frames. In these cases observationHandler won't be called. Observations need to be handled in completionHandler provided to VNCoreMLRequest.

let mlmodel = <#custom model#>
let vnmodel = try? VNCoreMLModel(for: mlmodel)
let request = VNCoreMLRequest(model: vnmodel!, completionHandler: <#observation Handler#>)

recognitionView.requests = [request]

Ordering and Filtering Observations

Because observations received in observationHandler of FUITextRecognitionViewController are not ordered, and finding relevant observations can become tedious, SAPMLTextObservationTopology provides APIs for finding adjacent observations, making it easy to filter observations.

First of all, let us understand the concepts of rows, columns, and blocks in a scanned image.

Rows, Columns, and Blocks

For illustration, a sample business card will be used.

Topology Example Image

SAPMLTextObservationTopology provides a grid-like topology of the observations that can be traversed easily to find relevant observations. The SAPMLTextObservationTopology instance can be created inside the observation handler using the following code:

recognitionView.observationHandler = { [weak self] observations in

    let topology = SAPMLTextObservationTopology(observations)

    return true
}

This topology instance has three key properties - rows, columns, and blocks. Each represents the observations in an ordered way. Listed below is the topology created for the sample business card.

Topology Example Image Values

Finding a Relevant Observation

In the business card, finding the Job Title in the second row becomes easy using the rows property of SAPMLTextObservationTopology.

recognitionView.observationHandler = { [weak self] observations in
    //creating topology of observations
    let topology = SAPMLTextObservationTopology(observations)

    //Second row contains the Job Title
    if (topology.rows.count >= 2)
    {
        let jobTitleObservation = topology.rows[1]
        recognitionView.showTexts(for: [jobTitleObservation], with: nil)
        DispatchQueue.main.asyncAfter(deadline: .now() + 0.8) {
            self.textField.text = (jobTitleObservation.value)
            self.dismiss(animated: true)
        }
        return true
    }
    return false
}

Accessing Adjacent Observations

previousObservationInLine, nextObservationInLine, previousObservationInColumn, and nextObservationInColumn methods of SAPMLTextObservationTopology help to find adjacent observations of a given SAPMLTextObservation.

For example, to fetch the state and ZIP code, which are to the left of the Email observation in the same line:

   let stateAndZipcodeObservation = topology.previousObservationInLine(emailObservation)
   // stateAndZipcodeObservation is "South Carolina, 29673" with comma as the delimiter
   let separatedArray = stateAndZipcodeObservation.componentsSeparatedByString(",")
   let stateObservation = separatedArray[0]
   let zipCodeObservation = separatedArray[1]

Setting Custom blockCreationDistance While Creating a Topology

It is possible to influence the block creation within the topology API. blockCreationDistance indicates how far away two observations can be placed from each other and still be considered to be in the same block. The default value is 0.01. The larger the value, the more observations that are far away will be considered to be in the same block.

   let topology = SAPMLTextObservationTopology(observations, blockCreationDistance: 0.1)

Managing and Updating ML Models

Core ML Models need to be updated frequently. However, pushing app updates to distribute an updated version of the ML Model is not necessary. SAPML provides SAPMLModelManager APIs to remotely download, update, and keep track of models used by the app.

Core ML Model files are uploaded as client resources on the SAP Mobile Services. The versioning support of client resources is essential to versioning of the Core ML Model files. SAPMLModelManager downloads the latest version of the model file hosted on the SAP Mobile Services.

Initialization and Configuration of SAPMLModelManager

Create an SAPMLModel object that provides information about the Core ML model. SAPMLModel encapsulates model properties, such as the name and URL of the model. SAPMLModelManager APIs take SAPMLModel instances to download and delete downloaded models. SAPMLModelManager also provides a list of downloaded SAPMLModel instances.

let helloCPMS = SAPMLModel(named:"HelloCPMSModel" , localURL: nil)

localURL is an optional parameter to use a local Core ML model as the initial version. Refer to Use a Local Core ML Model File as the Initial Version for additional information.

You can use SAPMLModelManager to download, update, delete, and list downloaded ML models. Before using SAPMLModelManager, all of its properties need to be set. SAPMLModelManager follows a singleton design pattern. To invoke SAPMLModelManager APIs, use an SAPMLModelManager.shared instance.

SAPMLModelManager.shared.sapUrlSession = //set the fully configured SAPURLSession
SAPMLModelManager.shared.cpmsSettingsParameters = //set the cpmsSettingsParameters
SAPMLModelManager.shared.delegate = MyMLModelManagerDelegate() //set the delegate that will receive callbacks on events

Usage Example

//Configure `SAPMLModelManager` by setting the `sapUrlSession` and `cpmsSettingsParameters` properties, which will be used for downloading the Core ML Models from SAP Mobile Services.
   SAPMLModelManager.shared.sapUrlSession = //set the fully configured SAPURLSession
   SAPMLModelManager.shared.cpmsSettingsParameters = //set the cpmsSettingsParameters
   SAPMLModelManager.shared.delegate = MyMLModelManagerDelegate() //set the delegate that will receive a callback on download completion

//Specify the name of the Core ML model that needs to be downloaded from SAP Mobile Services using SAPMLModel
   let helloCPMS = SAPMLModel(named:"HelloCPMSModel" , localURL: nil)

//Trigger download of the model
   SAPMLModelManager.shared.download(model: helloCPMS)

//Handle completion callback received on SAPMLModelManagerDelegate
func sapMLModelManager(_ manager: SAPMLModelManager, model: SAPMLModel?, didCompleteWithError error: SAPMLModelManagerError?) {
    if let err = error {
        switch err {
        case .compilationError(let compilationError):
            // Handle model compilation error
        case .downloadUnderway:
            // Handle model compilation error
        case .other(let otherError):
            // Handle otehr errors
        case .server(let serverError):
            // Handle server error
        case .cpmsSettingsMissing:
            // Handle CPMS Settings missing error
        case .modelNotFound:
            // Handle model not found error

        }
    }
    else {
        // Model downloaded/updated successfully
    }
}
//Access the downloaded and compiled ML model
    do {
       try SAPMLModelManager.shared.mlModel(for: helloCPMS)
    }
    catch _ {
       //Error Handling
    }

Note

SAPMLModelManager download and update complies with network synchronization policies specified by the SAP Mobile Services administrator.

Modifications Required in AppDelegate

Because ML models can be quite large in size, the ML model files are downloaded in the background. It is recommended to invoke download of the SAPMLModel in the AppDelegate applicationWillEnterForeground lifecycle method, so that the app gets to use the latest version.

func applicationWillEnterForeground(_: UIApplication) {
    let helloCPMS = SAPMLModel(named:"HelloCPMSModel" , localURL: nil)
    SAPMLModelManager.shared.download(model: helloCPMS)
}
//Because the `SAPMLModelManager` performs the download of Core ML models in the background, the `handleEventsForBackgroundURLSession` event must be forwarded to the SDK to handle completion of tasks when the application is launched or resumed in the background
func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
    AppDelegateDispatcher.application(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler)
}

Use a Local Core ML Model File as the Initial Version

There is an option to use a local Core ML Model file that is bundled with the app. This is treated as the initial version of the Core ML Model and will be served by the SAPMLModelManager, until a client resource file with the same name is uploaded to SAP Mobile Services. So, the application starts with a local Core ML model within the app, which is updated subsequently with the remotely distributed model from SAP Mobile Services.

//The Core ML model located at `localURL` will be available using the `mlModel(..)` function.
   let helloCPMS = SAPMLModel(named:"HelloCPMSModel" , localURL: Bundle.main.url(forResource: "myModel", withExtension: "mlmodelc"))
   SAPMLModelManager.shared.download(model: helloCPMS)
   try? SAPMLModelManager.shared.mlModel(for: helloCPMS) //returns local model

 //If a new version of the Core ML model is uploaded on SAP Mobile Services Client Resources, it will be downloaded and compiled.
   let helloCPMS = SAPMLModel(named:"HelloCPMSModel" , localURL: Bundle.main.url(forResource: "myModel", withExtension: "mlmodelc"))
   SAPMLModelManager.shared.download(model: helloCPMS)
   try? SAPMLModelManager.shared.mlModel(for: helloCPMS) //returns remote model

Last update: November 18, 2021