Build a Not Hotdog clone with React Native

If you're a fan of HBO's Silicon Valley, you'll remember when they launched a real AI-powered mobile app that classifies hotdogs from a given image (or not). Using Google's Vision API, let's try to recreate a working model of the application in React Native.

Google's Vision API is a machine learning tool that classifies details from an image provided as an input. The process of these classifications is based on thousands of different categories that are included in pre-trained API models. The Vision API enables access to these pre-trained models via a REST API.

• Prerequisites
• Setup Firebase Project
• Integrate Firebase SDK with React Native app
• Generate a Google Vision API Key
• Setting Permissions for Camera & Camera Roll
• Create a Header component
• Adding an Overlay Spinner
• Access Camera and Camera Roll
• Add functionality to determine a Hot dog
• Display final results
• Conclusion

To follow this tutorial, please make sure you have the following installed on your local development environment and have access to the services mentioned below:

• Node.js (>=10.x.x) with npm/yarn installed.
• expo-cli (>=3.0.9), previously known as create-react-native-app.
• a Google Cloud Platform account
• Firebase Storage setup
• Expo Client app for Android or iOS, used for testing the app

In this section, let us set up a new Firebase project. If you are already familiar with the process and know how to get a config keys from a Firebase project, you can skip this step.

Visit Firebase and sign-in with your Google ID. Once signed in, click on a new project and enter a name. Lastly, hit the Create project button.

After creating the project and being redirected to the dashboard screen, on the left side menu, click the settings icon, and then go to Project settings.

The whole the firebaseConfig object, as shown above, is required to integrate Firebase with a React Native or Expo app. Save them somewhere or make sure you know how to navigate to this page.

The next step is to setup Firebase storage rules such as to allow to upload image files through the app. From the left-hand side menu in the Firebase console, open Storage tab and then choose Rules. Modify them as follows.

Copy
service firebase.storage {
  match /b/{bucket}/o {
    match /{allPaths=**} {
      allow read, write
    }
  }
}

Firebase setup is complete.

To get started, create a new React Native project. For this demonstration, let us use expo-cli, an awesome tool that helps to create React Native apps at a faster rate. Open a terminal window, and run the following series of commands.

Copy
# generate a new app
expo init not-hotdog-app

# navigate inside the app folder
cd not-hotdog-app

# install the firebase SDK & other dependencies
yarn add firebase@6.0.1 expo-permissions
expo-image-picker uuid react-native-elements

Now that the project is generated open the directory in your favorite text editor. Then create a new folder called config and inside it, a new file called Firebase.js. This file will be responsible for integrating Firebase with the Expo app.

Copy
1import * as firebase from 'firebase';
2
3const firebaseConfig = {
4  apiKey: 'XXXX',
5  authDomain: 'XXXX',
6  databaseURL: 'XXXX',
7  projectId: 'XXXX',
8  storageBucket: 'XXXX',
9  messagingSenderId: 'XXXX',
10  appId: 'XXXX'
11};
12
13// Initialize Firebase
14firebase.initializeApp(firebaseConfig);
15
16export default firebase;

All the Xs are values of each key in the firebaseConfig object from the previous section. This completes the step to integrate a Firebase Web SDK with an Expo app.

Once you are signed in to Google Cloud Platform, you can visit the Google Cloud Console, to create a new project.

From the dropdown menu center, select a project. Then click the button New Project in the screen below. Notice you have already generated a Firebase project, select that from the list available.

Right now you are at the screen called Dashboard inside the console. From the top left, click on the menu button and a sidebar menu will pop up. Select APIs & Services > Dashboard.

At the Dashboard, select the button Enable APIs and Services.

Then search for the Vision API and make sure to click the button Enable.

Now, go back to the Dashboard and go to Credentials to generate an API key. Click the button Create Credentials and you will undergo a small process to generate the API key.

Once it is done, save the API key in App.js file after all the import statements.

Copy
1const VISION_API_KEY = 'XXXX';

The setup is complete. Let us move to the next section and start building the application.

To set permissions in any Expo app, all you need is to utilize an asynchronous method from the module expo-permissions. For this clone, there are two permissions that need to be set. The required permissions are for Camera and Camera Roll (or Photos of your device).

Camera roll is used in a case where the user wants to upload an image. For iOS simulator devs, you cannot access the camera so if you are not planning to use a real device until the end of this tutorial, but want to follow along. It is recommended to add Camera Roll functionality.

Import the permissions module in App.js file.

Copy
1import * as Permissions from 'expo-permissions';

Next step is to set an initial state that will control the View in the render method by determining whether the user has granted the permission to your app to use Camera and Camera roll or not.

Copy
1class App extends Component {
2  state = {
3    hasGrantedCameraPermission: false,
4    hasGrantedCameraRollPermission: false,
5  }

Next, using a lifecycle method componentDidMount(), define a promise for each permission. In the below snippet, you will find two functions cameraRollAccess() and cameraAccess() performing this operation. Respectively, each of these permission component has a permission type:

• for Camera Roll: Permissions.CAMERA_ROLL
• for Camera: Permissions.CAMERA

Copy
1async componentDidMount() {
2    this.cameraRollAccess()
3    this.cameraAccess()
4  }
5
6  cameraRollAccess = async () => {
7    const { status } = await Permissions.askAsync(Permissions.CAMERA_ROLL)
8
9    if (status === 'granted') {
10      this.setState({ hasGrantedCameraRollPermission: true })
11    }
12  }
13
14  cameraAccess = async () => {
15    const { status } = await Permissions.askAsync(Permissions.CAMERA)
16
17    if (status === 'granted') {
18      this.setState({ hasGrantedCameraPermission: true })
19    }
20  }

Each of the permission components returns a status value of granted or denied. In case of the permissions are granted, the value of state variables hasGrantedCameraRollPermission and hasGrantedCameraPermission are both set to true. The method Permissions.askAsync() to prompt the user for the type of permission.

Next, go to the render method of the App component and add condition using the two-state variables. If both are set to true, it will display the first screen of the application.

Copy
1 render() {
2    const {
3      hasGrantedCameraPermission,
4      hasGrantedCameraRollPermission,
5    } = this.state
6
7    if (
8      hasGrantedCameraPermission === false &&
9      hasGrantedCameraRollPermission === false
10    ) {
11      return (
12        <View style={{ flex: 1, marginTop: 100 }}>
13          <Text>No access to Camera or Gallery!</Text>
14        </View>
15      )
16    } else {
17      return (
18        <View style={styles.container}>
19          {*/ Rest of the content in the next section*/ }
20        </View>
21      )
22    }
23  }
24
25// Corresponding StyleSheet Object
26
27const styles = StyleSheet.create({
28  container: {
29    flex: 1,
30    backgroundColor: '#fff'
31  }
32})

If either or both are not granted, the app will display the message No access to Camera or Gallery!, also as shown below.

When tested on a real android device, it did ask for permissions.

Similarly, to use camera:

Using react-native-elements UI library for React Native, let us quickly create a useful header that will hold two buttons and the app's title in text. The left button will be to open the phone's gallery or camera roll consisting of user photos. The right button will be to open access the Camera on a real device.

Import the Header component from the react-native-elements library.

Copy
1import { View, Text, StyleSheet, TouchableOpacity } from 'react-native';
2import { Header, Icon } from 'react-native-elements';

The UI library has a pre-defined component called Header that you can use right away. This component accepts the icons on the left and right side. Since the app needs these icons to be clickable, use TouchableOpacity such that its prop can be later used to open the camera or the camera roll.

Copy
1<View style={styles.container}>
2  <Header
3    statusBarProps={{ barStyle: 'light-content' }}
4    backgroundColor="black"
5    leftComponent={
6      <TouchableOpacity onPress={() => alert('soon')}>
7        <Icon name="photo-album" color="#fff" />
8      </TouchableOpacity>
9    }
10    centerComponent={{
11      text: 'Not Hotdog?',
12      style: { color: '#fff', fontSize: 20, fontWeight: 'bold' }
13    }}
14    rightComponent={
15      <TouchableOpacity onPress={() => alert('soon')}>
16        <Icon name="camera-alt" color="#fff" />
17      </TouchableOpacity>
18    }
19  />
20</View>

The Header component also has a statusBarProps prop to change the color of the Status bar and works cross-platform. It will give the following output.

Both the icons are touchable, but right now they do not have an associated handler method except that a dummy alert message.

The react-native-elements library by default uses Material Icons and has a peer dependency of react-native-vector-icons.

The next element to add in the initial state object is uploading with a value of false. This variable will be used in the app to display an animated spinner whenever an image is being uploaded from the Camera Roll or analyzed by the Vision API for the result.

Copy
1state = {
2  //... rest,
3  uploading: false
4};
5
6// also make sure to include deconstruct the state inside render()
7const {
8  hasGrantedCameraPermission,
9  hasGrantedCameraRollPermission,
10  uploading
11} = this.state;

Create a new file inside components/UploadingOverlay.js. This file is going to contain a presentational component with the same name as the filename. Using ActivityIndicator from react-native you can animate this component by using its prop called animating.

Copy
1import React from 'react';
2import { ActivityIndicator, StyleSheet, View } from 'react-native';
3
4const UploadingOverlay = () => (
5  <View style={[StyleSheet.absoluteFill, styles.overlay]}>
6    <ActivityIndicator color="#000" animating size="large" />
7  </View>
8);
9
10const styles = StyleSheet.create({
11  overlay: {
12    backgroundColor: 'rgba(255,255,255,0.9)',
13    alignItems: 'center',
14    justifyContent: 'center'
15  }
16});
17
18export default UploadingOverlay;

Adding StyleSheet.absoluteFill to the style prop of the View component which holds the spinner, you can create an overlay screen. An overlay is just a screen or a View in terms of React Native that allows the current screen to appear on top of other screens. Using the backgroundColor property, you can add the opacity in the last after defining RBG values.

For example, when asking permission to access the Camera, a dialog box appeared on the app screen (as shown in the previous section). Notice how the box was position on top of the screen in the background.

Now, go back to App.js and add this component at the bottom of the render() section, just before the root View component is ending. Do not forget to import the component.

Copy
1import UploadingOverlay from './components/UploadingOverlay';
2
3// ... rest
4{
5  uploading ? <UploadingOverlay /> : null;
6}

The above condition states that, if the value of this.state.uploading is true, it will show the overlay screen. To test it out, temporarily set the value of uploading in the state object to true.

An endless spinner will continue to appear. Set the value of uploading back to false before proceeding.

In this section, you are going to add the functionality of accessing Camera and Camera Roll by defining three different handler functions in App component. Make sure you are inside the file App.js. First, import the following statement since this section is going to make use of Firebase's storage and uuid module to create a unique referent to each image.

Copy
1import firebase from './config/Firebase';
2import uuid from 'uuid';

Next, modify the initial state of the object to add the following for the final time.

Copy
1state = {
2  hasGrantedCameraPermission: false,
3  hasGrantedCameraRollPermission: false,
4  uploading: false,
5  image: null,
6  googleResponse: false
7};

To enable both of these functionalities in the current app, let us leverage another Expo module called expo-image-picker. First, import the module after the rest of the import statements.

Copy
1import * as ImagePicker from 'expo-image-picker';

Expo documentation has the best definition of what this module is used for. Take a look.

That's all you need right now. Define the first function, takePhoto that is going to access the phone's camera to click a photo.

Copy
1takePhoto = async () => {
2  let pickerResult = await ImagePicker.launchCameraAsync({
3    allowsEditing: true,
4    aspect: [4, 3]
5  });
6
7  this.handleImagePicked(pickerResult);
8};

The asynchronous method ImagePicker.launchCameraAsync() accepts two arguments:

• allowsEditing shows the UI to edit the image after it is clicked. Mostly used to crop images.
• aspect is an array to maintain a consistent aspect ratio if the allowsEditing is set to true.

Similarly, ImagePicker.launchImageLibraryAsync() is used with the same set of arguments to access Camera roll.

Copy
1pickImage = async () => {
2  let pickerResult = await ImagePicker.launchImageLibraryAsync({
3    allowsEditing: true,
4    aspect: [16, 9]
5  });
6
7  this.handleImagePicked(pickerResult);
8};

Both of these asynchronous functions, return the uri of the image selected (among other arguments that you can view in the official docs here). Lastly, both of these methods are calling another callback handleImagePicked after their job is done. This method contains the business of logic of how to handle the image after it is picked from the camera roll or clicked.

Copy
1handleImagePicked = async pickerResult => {
2  try {
3    this.setState({ uploading: true });
4
5    if (!pickerResult.cancelled) {
6      uploadUrl = await uploadImageAsync(pickerResult.uri);
7      this.setState({ image: uploadUrl });
8    }
9  } catch (e) {
10    console.log(e);
11    alert('Image Upload failed');
12  } finally {
13    this.setState({ uploading: false });
14  }
15};

Initially, set the state of uploading to true. Then, if an image is selected, call the custom method uploadImageAsync (which will be defined at the end of this section) and pass the URI of the image selected. This will also set the value of the image from the state object to the URL of the uploaded image. Lastly, set the state of the uploading in the finally block back to false if the results are positive and the image has uploaded without any errors.

The custom method uploadImageAsync has to be defined outside the App component. It will upload the image by creating a unique image ID or blob with the help of uuid. It uses xhr to make an Ajax call to send a request to the Firebase storage to upload the image.

Copy
1async function uploadImageAsync(uri) {
2  const blob = await new Promise((resolve, reject) => {
3    const xhr = new XMLHttpRequest();
4    xhr.onload = function () {
5      resolve(xhr.response);
6    };
7    xhr.onerror = function (e) {
8      console.log(e);
9      reject(new TypeError('Network request failed'));
10    };
11    xhr.responseType = 'blob';
12    xhr.open('GET', uri, true);
13    xhr.send(null);
14  });
15
16  const ref = firebase.storage().ref().child(uuid.v4());
17  const snapshot = await ref.put(blob);
18
19  blob.close();
20
21  return await snapshot.ref.getDownloadURL();
22}

Now you can add both the functions, pickImage and takePhoto as the value of onPress props for the corresponding icons.

Copy
1<Header
2  statusBarProps={{ barStyle: 'light-content' }}
3  backgroundColor="#000"
4  leftComponent={
5    <TouchableOpacity onPress={this.pickImage}>
6      <Icon name="photo-album" color="#fff" />
7    </TouchableOpacity>
8  }
9  centerComponent={{
10    text: 'Not Hotdog?',
11    style: styles.headerCenter
12  }}
13  rightComponent={
14    <TouchableOpacity onPress={this.takePhoto}>
15      <Icon name="camera-alt" color="#fff" />
16    </TouchableOpacity>
17  }
18/>

Here is an example of accessing Camera roll.

As most of the app is now set up, this section is going to be an interesting one. You are going to leverage the use of Google's Vision API to analyze whether the image provided by the user is a hot dog or not.

Inside the App component, add a new method called submitToGoogle. It is going to send requests and communicate with the API to fetch the result when a button is pressed by the user after the image has been uploaded. Again, while analyzing and fetching results, this method is going to set the state variable uploading to true. Then, it will send the URI of the image from the state object's image as the body of the request.

Along with the URI, the type of category you want to use is also defined along with a number of results it can fetch as a response. You can change the value of maxResults for the LABEL category. Currently, the value of the is set to 7. There are other detection categories provided by the Vision API other the one being used below, LABEL_DETECTION, such as a human face, logo, landmark, text, and so on.

Copy
1submitToGoogle = async () => {
2  try {
3    this.setState({ uploading: true });
4    let { image } = this.state;
5    let body = JSON.stringify({
6      requests: [
7        {
8          features: [{ type: 'LABEL_DETECTION', maxResults: 7 }],
9          image: {
10            source: {
11              imageUri: image
12            }
13          }
14        }
15      ]
16    });
17    let response = await fetch(
18      `https://vision.googleapis.com/v1/images:annotate?key=${VISION_API_KEY}`,
19      {
20        headers: {
21          Accept: 'application/json',
22          'Content-Type': 'application/json'
23        },
24        method: 'POST',
25        body: body
26      }
27    );
28    let responseJson = await response.json();
29    const getLabel = responseJson.responses[0].labelAnnotations.map(
30      obj => obj.description
31    );
32
33    let result =
34      getLabel.includes('Hot dog') ||
35      getLabel.includes('hot dog') ||
36      getLabel.includes('Hot dog bun');
37
38    this.setState({
39      googleResponse: result,
40      uploading: false
41    });
42  } catch (error) {
43    console.log(error);
44  }
45};

In the above snippet, the result is fetched in an array. Each array, in the current scenario, will have seven different objects. Using JavaScript's map let us extract the value of description from each object. All you need is to detect whether the description contains the word hotdog or not. This is done in the variable result. Lastly, the state of uploading overlay is set back to false, and the result of whether the uploaded image contains a hot dog or not is going to update googleResponse as boolean.

On a side note, the Vision API uses HTTP Post request as a REST API endpoint to perform data analysis on images you send in the request. This is done via the URL https://vision.googleapis.com/v1/images:annotate. To authenticate each request, you need the API key. The body of this POST request is in JSON format. For example:

Copy
1{
2  "requests": [
3    {
4      "image": {
5        "content": "/9j/7QBEUGhvdG9...image contents...eYxxxzj/Coa6Bax//Z"
6      },
7      "features": [
8        {
9          "type": "LABEL_DETECTION",
10          "maxResults": 1
11        }
12      ]
13    }
14  ]
15}

Using the boolean value from googleResponse, the end result is going to be output. The output will be displayed using renderImage.

Copy
1renderImage = () => {
2  let { image, googleResponse } = this.state;
3  if (!image) {
4    return (
5      <View style={styles.renderImageContainer}>
6        <Button
7          buttonStyle={styles.button}
8          onPress={() => this.submitToGoogle()}
9          title="Check"
10          titleStyle={styles.buttonTitle}
11          disabled
12        />
13        <View style={styles.imageContainer}>
14          <Text style={styles.title}>Upload an image to verify a hotdog!</Text>
15          <Text style={styles.hotdogEmoji}>🌭</Text>
16        </View>
17      </View>
18    );
19  }
20  return (
21    <View style={styles.renderImageContainer}>
22      <Button
23        buttonStyle={styles.button}
24        onPress={() => this.submitToGoogle()}
25        title="Check"
26        titleStyle={styles.buttonTitle}
27      />
28
29      <View style={styles.imageContainer}>
30        <Image source={{ uri: image }} style={styles.imageDisplay} />
31      </View>
32
33      {googleResponse ? (
34        <Text style={styles.hotdogEmoji}>🌭</Text>
35      ) : (
36        <Text style={styles.hotdogEmoji}>❌</Text>
37      )}
38    </View>
39  );
40};

The Button component used above is from react-native-elements library. It is going to be disabled until no image is selected. On its prop onPress the handle function submitToGoogle is called. The second view displays the image, and beneath it, an emoji is showcased whether the image has the desired result or not. Do note that by default the cross emoji will be showcased since the default value of googleResponse is set to false when defining the initial state. Only after clicking the button, the emoji displayed is the final result.

Lastly, do not forget to add renderImage inside App component's render method, just before the UploadingOverlay component.

Copy
1// inside the render method
2{
3  this.renderImage();
4}
5{
6  uploading ? <UploadingOverlay /> : null;
7}

Here is a short demo of how the app looks and works on a real android device using Expo client to run the app.

Here is complete source code for StyleSheet object.

Copy
1const styles = StyleSheet.create({
2  container: {
3    flex: 1,
4    backgroundColor: '#cafafe'
5  },
6  headerCenter: {
7    color: '#fff',
8    fontSize: 20,
9    fontWeight: 'bold'
10  },
11  renderImageContainer: {
12    marginTop: 20,
13    alignItems: 'center'
14  },
15  button: {
16    backgroundColor: '#97caef',
17    borderRadius: 10,
18    width: 150,
19    height: 50
20  },
21  buttonTitle: {
22    fontWeight: '600'
23  },
24  imageContainer: {
25    margin: 25,
26    alignItems: 'center'
27  },
28  imageDisplay: {
29    width: 300,
30    height: 300
31  },
32  title: {
33    fontSize: 36
34  },
35  hotdogEmoji: {
36    marginTop: 20,
37    fontSize: 90
38  }
39});
40
41export default App;

If you visit the storage section in Firebase, you can notice that each image is stored with a name of base64 binary string.

By integrating Firebase storage and using Google's Vision API with React Native, you have completed this tutorial. The API is amazing with endless use cases. I hope you learned a thing or two by reading this post. The complete source code for this app is available at this Github repo. Some of the resources used in this post:

• react-native-elements UI component library
• expo-image-picker
• firebase-storage-upload-example with expo

Originally published at Heartbeat