Building AR face masks in React Native

Augmented reality (AR) is a technology that blends computer-generated images with the real world. Building AR applications with React Native has become popular due to its cross-platform capabilities, vast community support, and a variety of libraries that enable developers to create immersive AR experiences.

This article will review how to build a custom AR face mask using DeepAR and use the face mask in a React Native application. See the full code for this post here on GitHub.

Prerequisites

This tutorial assumes a basic understanding of React Native and the necessary development tools. Knowledge of DeepAR is not required.

We’ll build a simple React Native app that loads an AR face mask from DeepAR, as shown:

Options for React Native AR frameworks

There are several AR frameworks that you can choose from to add AR filters and effects to your React Native application. Some options include:

• DeepAR:
  • Provides tools and SDKs for you to create AR filters and effects and add them to any platform, accompanied by well-written documentation
  • DeepAR is priced, and the free tier is only limited to a certain number of users
  • DeepAR projects can be added to your React Native app using the react-native-deepar package
• ViroReact:
  • Open-source community platform that allows you to create AR/VR applications for free
  • ViroReact supports ARKit and ARCore
  • It’s well-documented, easy to learn, and can be used on any platform
• Banuba:
  • Offers SDKs for you to add AR filters and effects to your app
  • The Face AR SDK is supported on various platforms and is well-documented, but you need to book a free trial to get an SDK token
  • Pricing for the AR SDK is not disclosed on their site

These are just a few of the options available. In this article, we’ll use the DeepAR platform to create, edit, and add an AR face mask to our React Native application.

What is DeepAR?

DeepAR is a framework for creating augmented reality applications. DeepAR offers various products that allow developers to create immersive experiences with AR, such as:

• DeepAR SDK: Allows you to integrate AR into any application
• DeepAR Studio: Allows you to create AR filters and effects
• Asset Store: Allows you to try ready-made filters and effects

Some of the key DeepAR features include face filters, effects, masks, AR beauty and makeup filters, Realtime Emotion Detection, and more. Though not free, DeepAR allows you to create free applications that can be used by up to 10 people.

Getting started with DeepAR

First, you’ll need to create a DeepAR Account and then create a new free project:

Next, you’ll be required to link an app. We’ll create an Android and iOS app. Add the desired app and bundle ID/app ID, which will also be updated in our Android and iOS React Native apps to match:

You should be provided with a license key, which we’ll use to deploy our DeepAR apps:

Create a custom AR face mask

We’ll use the DeepAR Studio to build and render a custom face mask in our React Native app. You can use it to create or test customized AR filters.

First, install DeepAR Studio directly from the DeepAR Developer Portal. Once the installation setup is complete, create a new project:

You’ll be directed to the editor, where you’ll create your AR face mask:

Next, click the Root section under the hierarchy editor then click Add component. Select Mesh Render as your component:

Then, set your mesh to Face and on the material select Duplicate. This will add default material under Assets:

Clicking on the material added allows you to modify different properties of the face mask:

We’ll add a custom texture to the face mesh. A texture is an image or a painting that you want to place on top of the model/mesh.

First, import your custom texture by clicking the Add button under Assets. Import your resources. This will add the resources under textures:

You’ll then add the new texture as the surface material of the AR face mask. Click the material you added, then set the Shader to MatCap:

Then, set the color texture to the imported texture. This paints the face mask according to the texture you added:

Once you’re done setting your desired properties, you can open a live preview of your mask:

If you’re satisfied with the output, export the face mask. The file should be saved with a .deepar file extension:

Since we intend to store the filter in online storage and fetch it over the internet, we’ll use Supabase to store the asset. You can use any storage platform you want for the asset if Supabase isn’t for you; you might even want to go ahead and use the asset locally in your React Native application.

If you do use Supabase, create public storage, then add the exported file. You can then use the URL you obtain to fetch the asset in your app.

With that, you’re ready to use your face mask in your React Native application.

Using the AR face mask in React Native

First, create a React Native application using the command below:

npx react-native@latest init reactdeepar

Next, add the following dependencies in your package.json file, then run Yarn to install them:

"react-native-deepar": "^0.11.0",
"rn-fetch-blob": "^0.12.0"

The react-native-deepar package offers a React Native wrapper for DeepAR, and we’ll use the rn-fetch-blob package to fetch our custom face mask over the internet.

Update your App.tsx file with the code snippet below. Replace <YOUR-SUPABASE-FILE-URL> with the URL of the file stored on Supabase and <YOUR-API-KEY> with your app-specific DeepAR license key:

// Add imports
import {useEffect, useRef, useState} from 'react';
import {
  Dimensions,
  Linking,
  SafeAreaView,
  StyleSheet,
  Text,
  TouchableOpacity,
  View,
} from 'react-native';
import DeepARView, {
  IDeepARHandle,
  Camera,
  CameraPermissionRequestResult,
  ErrorTypes,
  CameraPositions,
} from 'react-native-deepar';
import RNFetchBlob from 'rn-fetch-blob';

const App = () => {
  const deepARRef = useRef<IDeepARHandle>(null);
  const [permsGranted, setPermsGranted] = useState(false);
  const getPermissions = async () => {
    const cameraPermission = await Camera.requestCameraPermission();
    const isCameraAllowed =
      cameraPermission === CameraPermissionRequestResult.AUTHORIZED;
    if (isCameraAllowed) {
      setPermsGranted(true);
    } else {
      Linking.openSettings();
    }
  };
  useEffect(() => {
    getPermissions();
  }, []);
  const loadEffect = () => {
    RNFetchBlob.config({
      fileCache: true,
    })
      .fetch('GET', '<YOUR-SUPABASE-FILE-URL>')
      .then(res => {
        deepARRef?.current?.switchEffectWithPath({
          path: res.path(),
          slot: 'mask',
        });
      });
  };
  const renderPhotoButton = () => {
    return (
      <>
        <View style={styles.button}>
          <TouchableOpacity onPress={loadEffect} style={styles.effectButton}>
            <Text style={styles.effectText}>Load Effect</Text>
          </TouchableOpacity>
        </View>
      </>
    );
  };
  const renderDeepARView = () => {
    if (permsGranted === false) {
      return null;
    }
    return (
      <>
        <DeepARView
          ref={deepARRef}
          apiKey="<YOUR-API-KEY>"
          videoWarmup={false}
          position={CameraPositions.FRONT}
          style={styles.deepARView}
          onError={(text: String, type: ErrorTypes) => {
            console.log('onError =>', text, 'type =>', type);
          }}
        />
        {renderPhotoButton()}
      </>
    );
  };
  return (
    <SafeAreaView>
      <View style={styles.container}>{renderDeepARView()}</View>
    </SafeAreaView>
  );
};
export default App;

This code does the following:

• Adds the necessary imports to our project
• Creates a reference for your DeepAR component and adds the app’s permission state
• Adds the logic to the request and sets the necessary camera permissions
• Fetches the DeepAR face mask from Supabase storage, and then caches the result in a reachable folder using the switchEffectWithPath method from DeepAR. This method expects a path where the asset will be stored, and a slot, which is a unique name for your face mask
• Renders the DeepARView with your API key. You can display the face mask by clicking the button in the UI

Using the face mask

To run the project on Android or iOS, follow the steps in this article’s GitHub repository. When you run your application and load the effect, your AR face mask displays as shown below:

Conclusion

In this tutorial, we discussed the various options for AR in your React Native application. We explored DeepAR and some of its products and features. We then created a custom DeepAR face mask using DeepAR studio and used the face mask in our React Native app. The complete code used in this article is available on GitHub.

The face mask built in this article is simple and standard, but DeepAR is flexible and allows you to create more complex masks. Hope you found this article helpful, happy coding!