Image Classification on React Native with TensorFlow.js and MobileNet

Published on Oct 17, 2019

•

10 min read

•

EXPO

Recently, the alpha version Tensorflow.js for React Native and Expo applications was released. It currently provides the capabilities of loading pre-trained models and training. Here is the announcement tweet:

Tweet

TensorFlow.js provides many pre-trained models that simplify the time-consuming task of training a machine learning model from scratch. In this tutorial, we are going to explore Tensorflow.js and the MobileNet pre-trained model to classify image based on the input image provided in a React Native mobile application.

Here is the link to the complete code in a Github repo for your reference.

Requirements

🔗

Nodejs >= 10.x.x install on your local dev environment
expo-cli
Expo Client app for Android or iOS, used for testing the app

Integrating TFJS in an Expo app

🔗

To start and use the Tensorflow library in a React Native application, the initial step is to integrate the platform adapter. The module tfjs-react-native is the platform adapter that supports loading all major tfjs models from the web. It also provides GPU support using expo-gl.

Open the terminal window, and create a new Expo app by executing the command below.


expo init mobilenet-tfjs-expo

Next, make sure to generate Expo managed app. Then navigate inside the app directory and install the following dependencies.


yarn add @react-native-community/async-storage
@tensorflow/tfjs @tensorflow/tfjs-react-native
expo-gl @tensorflow-models/mobilenet jpeg-js

Note: If you are looking forward to using react-native-cli to generate an app, you can follow the clear instructions to modify metro.config.js file and other necessary steps, mentioned here.

Even though you are using Expo, it is necessary to install async-storage as tfjs module depends on that.

Testing TFJS that it is working

🔗

Before we move on, let us test out that the tfjs is getting loaded into the app before the app is rendered. There is an asynchronous function to do so, called tf.ready(). Open App.js file, import the necessary dependencies, and define an initial state isTfReady with a boolean false.


1import React from 'react';
2import { StyleSheet, Text, View } from 'react-native';
3import * as tf from '@tensorflow/tfjs';
4import { fetch } from '@tensorflow/tfjs-react-native';
5
6class App extends React.Component {
7  state = {
8    isTfReady: false
9  };
10
11  async componentDidMount() {
12    await tf.ready();
13    this.setState({
14      isTfReady: true
15    });
16
17    //Output in Expo console
18    console.log(this.state.isTfReady);
19  }
20
21  render() {
22    return (
23      <View style={styles.container}>
24        <Text>TFJS ready? {this.state.isTfReady ? <Text>Yes</Text> : ''}</Text>
25      </View>
26    );
27  }
28}
29
30const styles = StyleSheet.create({
31  container: {
32    flex: 1,
33    backgroundColor: '#fff',
34    alignItems: 'center',
35    justifyContent: 'center'
36  }
37});
38
39export default App;

Since the lifecycle method is asynchronous, it will only update the value of isTfReady to true when tfjs is actually loaded.

You can see the output in the simulator device as shown below.

Or in the console, if using the console statement as the above snippet.

Loading Tensorflow model

🔗

Similar to the previous section, you can load the model being used in this app (mobilenet) is integrating or not. Loading a tfjs pre-trained model from the web is an expensive network call and will take a good amount of time. Modify the App.js file to load the MobileNet model. Start by importing the model.


1import * as mobilenet from '@tensorflow-models/mobilenet';

Next, add another property to the initial state.


1state = {
2  isTfReady: false,
3  isModelReady: false
4};

Then, modify the lifecycle method.


1async componentDidMount() {
2    await tf.ready()
3    this.setState({
4      isTfReady: true
5    })
6    this.model = await mobilenet.load()
7    this.setState({ isModelReady: true })
8}

Lastly, the display on the screen when the loading of the model is complete.


1<Text>
2  Model ready?{' '}
3  {this.state.isModelReady ? <Text>Yes</Text> : <Text>Loading Model...</Text>}
4</Text>

When the model is being loaded, it will display the following message.

When the loading of the MobileNet model is complete, you will get the following output.

Asking user permissions

🔗

Now that both the platform adapter and the model are currently integrated with the React Native app, add an asynchronous function to ask for the user's permission to allow access to the camera roll. This is a mandatory step when building iOS applications using the image picker component from Expo.

Before, you proceed, run the following command to install all the packages provided by Expo SDK.


expo install expo-permissions expo-constants expo-image-picker

Next, add the following import statements in the App.js file.


1import Constants from 'expo-constants';
2import * as Permissions from 'expo-permissions';

In the App class component, add the following method.


1getPermissionAsync = async () => {
2  if (Constants.platform.ios) {
3    const { status } = await Permissions.askAsync(Permissions.CAMERA_ROLL);
4    if (status !== 'granted') {
5      alert('Sorry, we need camera roll permissions to make this work!');
6    }
7  }
8};

Lastly, call this asynchronous method inside componentDidMount().


1async componentDidMount() {
2    await tf.ready()
3    this.setState({
4      isTfReady: true
5    })
6    this.model = await mobilenet.load()
7    this.setState({ isModelReady: true })
8
9    // add this
10    this.getPermissionAsync()
11  }

Convert a raw image into a Tensor

🔗

The application will require the user to upload an image from their phone's camera roll or gallery. You have to add a handler method that is going to load the image and allow the Tensorflow to decode the data from the image. Tensorflow supports JPEG and PNG formats.

In the App.js file, start by importing jpeg-js package that will be used to decode the data from the image.


1import * as jpeg from 'jpeg-js';

It decodes the width, height and the binary data from the image inside the handler method imageToTensor that accepts a parameter of the raw image data.


1imageToTensor(rawImageData) {
2    const TO_UINT8ARRAY = true
3    const { width, height, data } = jpeg.decode(rawImageData, TO_UINT8ARRAY)
4    // Drop the alpha channel info for mobilenet
5    const buffer = new Uint8Array(width * height * 3)
6    let offset = 0 // offset into original data
7    for (let i = 0; i < buffer.length; i += 3) {
8      buffer[i] = data[offset]
9      buffer[i + 1] = data[offset + 1]
10      buffer[i + 2] = data[offset + 2]
11
12      offset += 4
13    }
14
15    return tf.tensor3d(buffer, [height, width, 3])
16  }

The TO_UINT8ARRAY array represents an array of 8-bit unsigned integers. the constructor method Uint8Array() is the new ES2017 syntax. There are different types of typed arrays, each having its own byte range in the memory.

Load and Classify the image

🔗

Next, we add another handler method classifyImage that will read the raw data from an image and yield results upon classification in the form of predictions.

The image is going to be read from a source and the path to that image source has to be saved in the state of the app component. Similarly, the results yield by this asynchronous method have to be saved too. Modify the existing state in the App.js file for the final time.


1state = {
2  isTfReady: false,
3  isModelReady: false,
4  predictions: null,
5  image: null
6};

Next, add the asynchronous method.


1classifyImage = async () => {
2  try {
3    const imageAssetPath = Image.resolveAssetSource(this.state.image);
4    const response = await fetch(imageAssetPath.uri, {}, { isBinary: true });
5    const rawImageData = await response.arrayBuffer();
6    const imageTensor = this.imageToTensor(rawImageData);
7    const predictions = await this.model.classify(imageTensor);
8    this.setState({ predictions });
9    console.log(predictions);
10  } catch (error) {
11    console.log(error);
12  }
13};

The results from the pre-trained model are yield in an array. An example is shown below.

Allow user to pick the image

🔗

To select an image from the device's camera roll using the system's UI, you are going to use the asynchronous method ImagePicker.launchImageLibraryAsync provided the package expo-image-picker. Import the package itself.


1import * as Permissions from 'expo-permissions';

Next, add a handler method selectImage that will be responsible for

let the image to be selected by the user
if the image selection process is not canceled, populate the source URI object in the state.image
lastly, invoke classifyImage() method to make predictions from the given input


1selectImage = async () => {
2  try {
3    let response = await ImagePicker.launchImageLibraryAsync({
4      mediaTypes: ImagePicker.MediaTypeOptions.All,
5      allowsEditing: true,
6      aspect: [4, 3]
7    });
8
9    if (!response.cancelled) {
10      const source = { uri: response.uri };
11      this.setState({ image: source });
12      this.classifyImage();
13    }
14  } catch (error) {
15    console.log(error);
16  }
17};

The package expo-image-picker returns an object. In case the user cancels the process of picking an image, the image picker module will return a single property: canceled: true. f successful, the image picker module returns properties such as the uri of the image itself. That’s why the if statement in the above snippet holds so much significance.

Run the app

🔗

To complete this demonstration app, you need to add a touchable opacity where the user will click to add the image.

Here is the complete snippet of the render method in the App.js file.


1render() {
2    const { isTfReady, isModelReady, predictions, image } = this.state
3
4    return (
5      <View style={styles.container}>
6        <StatusBar barStyle='light-content' />
7        <View style={styles.loadingContainer}>
8          <Text style={styles.commonTextStyles}>
9            TFJS ready? {isTfReady ? <Text>✅</Text> : ''}
10          </Text>
11
12          <View style={styles.loadingModelContainer}>
13            <Text style={styles.text}>Model ready? </Text>
14            {isModelReady ? (
15              <Text style={styles.text}>✅</Text>
16            ) : (
17              <ActivityIndicator size='small' />
18            )}
19          </View>
20        </View>
21        <TouchableOpacity
22          style={styles.imageWrapper}
23          onPress={isModelReady ? this.selectImage : undefined}>
24          {image && <Image source={image} style={styles.imageContainer} />}
25
26          {isModelReady && !image && (
27            <Text style={styles.transparentText}>Tap to choose image</Text>
28          )}
29        </TouchableOpacity>
30        <View style={styles.predictionWrapper}>
31          {isModelReady && image && (
32            <Text style={styles.text}>
33              Predictions: {predictions ? '' : 'Predicting...'}
34            </Text>
35          )}
36          {isModelReady &&
37            predictions &&
38            predictions.map(p => this.renderPrediction(p))}
39        </View>
40        <View style={styles.footer}>
41          <Text style={styles.poweredBy}>Powered by:</Text>
42          <Image source={require('./assets/tfjs.jpg')} style={styles.tfLogo} />
43        </View>
44      </View>
45    )
46  }
47}

Here is the list of the complete styles object.


1const styles = StyleSheet.create({
2  container: {
3    flex: 1,
4    backgroundColor: '#171f24',
5    alignItems: 'center'
6  },
7  loadingContainer: {
8    marginTop: 80,
9    justifyContent: 'center'
10  },
11  text: {
12    color: '#ffffff',
13    fontSize: 16
14  },
15  loadingModelContainer: {
16    flexDirection: 'row',
17    marginTop: 10
18  },
19  imageWrapper: {
20    width: 280,
21    height: 280,
22    padding: 10,
23    borderColor: '#cf667f',
24    borderWidth: 5,
25    borderStyle: 'dashed',
26    marginTop: 40,
27    marginBottom: 10,
28    position: 'relative',
29    justifyContent: 'center',
30    alignItems: 'center'
31  },
32  imageContainer: {
33    width: 250,
34    height: 250,
35    position: 'absolute',
36    top: 10,
37    left: 10,
38    bottom: 10,
39    right: 10
40  },
41  predictionWrapper: {
42    height: 100,
43    width: '100%',
44    flexDirection: 'column',
45    alignItems: 'center'
46  },
47  transparentText: {
48    color: '#ffffff',
49    opacity: 0.7
50  },
51  footer: {
52    marginTop: 40
53  },
54  poweredBy: {
55    fontSize: 20,
56    color: '#e69e34',
57    marginBottom: 6
58  },
59  tfLogo: {
60    width: 125,
61    height: 70
62  }
63});

Run the application by executing the expo start command from a terminal window. The first thing you’ll notice is that upon bootstrapping the app in the Expo client, it will ask for permissions.

Then, once the model is ready, it will display the text "Tap to choose image" inside the box. Select an image to see the results.

Predicting results can take some time. Here are the results of the previously selected image.

Conclusion

🔗

I hope this post serves the purpose of giving you a head start in understanding how to implement a TesnorFlow.js model in a React Native app, as well as a better understanding of image classification, a core use case in computer vision-based machine learning.

Since the TF.js for React Native is in alpha at the time of writing this post, we can hope to see many more advanced examples in the future to build real-time applications. Here are some resources that I find extremely useful.

Here are some resources that I find extremely useful.

tfjs-react-native Github repo contain more examples using different pre-trained models
Infinite Red's NSFW JS and React Native example
Introduction to Tensorflow.js

You can find the complete code at this Github repo.

Originally published at Heartbeat.Fritz.ai

Share this article on X (Twitter)

Browse all posts

⬅️ Previous: How to implement Forgot Password feature in React Native with Firebase

Next: Build and validate forms in React Native using Formik and Yup ➡️

Aman Mittal author

Buy me coffee

GitHub

X (Twitter)

I'm a software developer and a technical writer. On this blog, I write about my learnings in software development and technical writing.

Currently, working maintaining docs at 𝝠 Expo. Read more about me on the About page.