Create a React Native Image Recognition App with Google Vision API

2019-02-13

Introduction

Google Cloud Vision API is a machine learning tool that can classify details from an image provided as an input into thousands of different categories with pre-trained API models. It offers these pre-trained models through an API and the categories are detected as individual objects within the image. In this tutorial, you are going to learn how to integrate Google Cloud Vision API in a React Native application and make use of real time APIs.

Installing Expo

If you are not familiar with Expo, this tutorial can be a good start. Basically, Expo provides a set of tools to create and publish React Native applications with minimal effort. Earlier, React Native had something called create-react-native-app which is now merged with Expo-Cli and is an official way to build a React Native app. To create your React Native app, you need to install Expo as a global npm module.

1npm install -g expo-cli

Once the command line interface for Expo is installed in your local development environment, you must run the following command in order to generate a project.

1expo-cli init google-vision-rn-demo

It will ask you for which template to use; choose the option blank template rather than tabs template. We only need a single screen in our application for the demonstration purposes. In the last step, you will be prompted to write the name of the project - simply type it and hit enter. Then, it will start installing dependencies. Once the project is created, traverse into the project directory. If you need any help with this setup, refer to the Expo documentation.

Setting Up Firebase

In this section, we are going to set up a new Firebase project. It will provide us the database and backend service and we do not have to write our own backend for this tutorial, hence saving time and focusing on what we need to learn. For simplicity, I am going to make the Firebase project data public for demonstration purposes.

Visit Firebase and sign-in with your Google ID. Once signed-in, click on a new project and enter a name. Lastly, hit the Create Project button.

ss1

The next step is to make sure we set up Firebase database rules to allow us to upload image files through the app. From the left-hand side menu in the Firebase console, open Database tab and then choose Rules and modify them as follows.

1service cloud.firestore {
2 match /databases/{database}/documents {
3 match /{document=**} {
4 allow read, write;
5 }
6 }
7}

We need to install the Firebase SDK in our React Native app. Run the following command from your terminal.

1npm install -S firebase

Now, create a folder called config and inside it, create a new file called environment.js. This file will contain all keys needed to bootstrap and hook Firebase with our application.

1//environment.js
2var environments = {
3 staging: {
4 FIREBASE_API_KEY: 'XXXX',
5 FIREBASE_AUTH_DOMAIN: 'XXXX',
6 FIREBASE_DATABASE_URL: 'XXXX',
7 FIREBASE_PROJECT_ID: 'XXXX',
8 FIREBASE_STORAGE_BUCKET: 'XXXX',
9 FIREBASE_MESSAGING_SENDER_ID: 'XXXX',
10 GOOGLE_CLOUD_VISION_API_KEY: 'XXXX'
11 },
12 production: {
13 // Warning: This file still gets included in your native binary and is not a secure way to store secrets if you build for the app stores. Details: https://github.com/expo/expo/issues/83
14 }
15}
16
17function getReleaseChannel() {
18 let releaseChannel = Expo.Constants.manifest.releaseChannel
19 if (releaseChannel === undefined) {
20 return 'staging'
21 } else if (releaseChannel === 'staging') {
22 return 'staging'
23 } else {
24 return 'staging'
25 }
26}
27function getEnvironment(env) {
28 console.log('Release Channel: ', getReleaseChannel())
29 return environments[env]
30}
31var Environment = getEnvironment(getReleaseChannel())
32export default Environment

The Xs are values of each key you have to fill in. Ignore the value for Key GOOGLE_CLOUD_VISION_API_KEY right now as we will get back to it in the next section. Other values for their corresponding keys can be attained at the Firebase console. You can get these values by visiting Firebase console and then click the gear icon next to Project Overview in the left-hand side menu bar and lastly go to Project settings section. There are ways in Expo where you do not have to publish your secret keys when deploying the app or uploading the codebase on a site like Github. The initial step I would recommend is to add this file inside .gitignore.

Then create another file called firebase.js inside the config directory. We will be using this file in the main application to send requests to upload an image to the Firebase storage. Also note that we are importing environment.js in it to access Firebase keys.

1// firebase.js
2import * as firebase from 'firebase'
3
4firebase.initializeApp({
5 apiKey: Environment['FIREBASE_API_KEY'],
6 authDomain: Environment['FIREBASE_AUTH_DOMAIN'],
7 databaseURL: Environment['FIREBASE_DATABASE_URL'],
8 projectId: Environment['FIREBASE_PROJECT_ID'],
9 storageBucket: Environment['FIREBASE_STORAGE_BUCKET'],
10 messagingSenderId: Environment['FIREBASE_MESSAGING_SENDER_ID']
11})
12
13export default firebase

Getting Google Cloud Vision API Key

To use a Google Cloud Platform service, you need a Gmail account. Once you are signed-in from your Gmail ID, you can visit the Google Cloud Console. The next step is to create a new project.

ss6

Click select a project from the drop-down menu and then click new project. Enter the name of your project and then click Create. Once you’ve created the project, we are placed back into the main console page again and then need to select our newly created project.

The next step in this process is to get your API key. This you can get by clicking on the console and moving over to Dashboard section and under that choose Enable APIs and Services.

ss7

Then type vision in the search on the page as shown below.

ss8

And then click Vision API.

ss9

Lastly, click Enable like below

ss10

In order to complete this process of enabling Vision API services, you are required to add billing information (if you haven't done already) to your Google Cloud Platform account.

Your URL in the dashboard will look like this: https://console.cloud.google.com/apis/dashboard?project=FIREBASE-PROJECT-ID&folder&organizationId. Once you are at the below screen, click on the Credentials section from the left-hand side menu and create a new API key if there isn't any by clicking on the button Create Credentials and then API Key.

ss2

Once you have created your API key, it is time to add it in the file environment.js for the key GOOGLE_CLOUD_VISION_API_KEY.

That's it. Setting up the APIs is complete. We can now move on to work on the app itself.

Building The App

To get started, we need to install an npm package called uuid to create a unique blob for the image that is going to upload on the Firebase storage service. Run the command npm install --save uuid. Next, open App.js and paste the following code.

1import React from 'react'
2import {
3 ActivityIndicator,
4 Button,
5 Clipboard,
6 FlatList,
7 Image,
8 Share,
9 StyleSheet,
10 Text,
11 ScrollView,
12 View
13} from 'react-native'
14import { ImagePicker, Permissions } from 'expo'
15import uuid from 'uuid'
16import Environment from './config/environment'
17import firebase from './config/firebase'
18
19export default class App extends React.Component {
20 state = {
21 image: null,
22 uploading: false,
23 googleResponse: null
24 }
25
26 async componentDidMount() {
27 await Permissions.askAsync(Permissions.CAMERA_ROLL)
28 await Permissions.askAsync(Permissions.CAMERA)
29 }
30
31 render() {
32 let { image } = this.state
33
34 return (
35 <View style={styles.container}>
36 <ScrollView
37 style={styles.container}
38 contentContainerStyle={styles.contentContainer}
39 >
40 <View style={styles.getStartedContainer}>
41 {image ? null : (
42 <Text style={styles.getStartedText}>Google Cloud Vision</Text>
43 )}
44 </View>
45
46 <View style={styles.helpContainer}>
47 <Button
48 onPress={this._pickImage}
49 title="Pick an image from camera roll"
50 />
51
52 <Button onPress={this._takePhoto} title="Take a photo" />
53 {this.state.googleResponse && (
54 <FlatList
55 data={this.state.googleResponse.responses[0].labelAnnotations}
56 extraData={this.state}
57 keyExtractor={this._keyExtractor}
58 renderItem={({ item }) => <Text>Item: {item.description}</Text>}
59 />
60 )}
61 {this._maybeRenderImage()}
62 {this._maybeRenderUploadingOverlay()}
63 </View>
64 </ScrollView>
65 </View>
66 )
67 }
68
69 organize = array => {
70 return array.map(function(item, i) {
71 return (
72 <View key={i}>
73 <Text>{item}</Text>
74 </View>
75 )
76 })
77 }
78
79 _maybeRenderUploadingOverlay = () => {
80 if (this.state.uploading) {
81 return (
82 <View
83 style={[
84 StyleSheet.absoluteFill,
85 {
86 backgroundColor: 'rgba(0,0,0,0.4)',
87 alignItems: 'center',
88 justifyContent: 'center'
89 }
90 ]}
91 >
92 <ActivityIndicator color="#fff" animating size="large" />
93 </View>
94 )
95 }
96 }
97
98 _maybeRenderImage = () => {
99 let { image, googleResponse } = this.state
100 if (!image) {
101 return
102 }
103
104 return (
105 <View
106 style={{
107 marginTop: 20,
108 width: 250,
109 borderRadius: 3,
110 elevation: 2
111 }}
112 >
113 <Button
114 style={{ marginBottom: 10 }}
115 onPress={() => this.submitToGoogle()}
116 title="Analyze!"
117 />
118
119 <View
120 style={{
121 borderTopRightRadius: 3,
122 borderTopLeftRadius: 3,
123 shadowColor: 'rgba(0,0,0,1)',
124 shadowOpacity: 0.2,
125 shadowOffset: { width: 4, height: 4 },
126 shadowRadius: 5,
127 overflow: 'hidden'
128 }}
129 >
130 <Image source={{ uri: image }} style={{ width: 250, height: 250 }} />
131 </View>
132 <Text
133 onPress={this._copyToClipboard}
134 onLongPress={this._share}
135 style={{ paddingVertical: 10, paddingHorizontal: 10 }}
136 />
137
138 <Text>Raw JSON:</Text>
139
140 {googleResponse && (
141 <Text
142 onPress={this._copyToClipboard}
143 onLongPress={this._share}
144 style={{ paddingVertical: 10, paddingHorizontal: 10 }}
145 >
146 JSON.stringify(googleResponse.responses)}
147 </Text>
148 )}
149 </View>
150 )
151 }
152
153 _keyExtractor = (item, index) => item.id
154
155 _renderItem = item => {
156 ;<Text>response: {JSON.stringify(item)}</Text>
157 }
158
159 _share = () => {
160 Share.share({
161 message: JSON.stringify(this.state.googleResponse.responses),
162 title: 'Check it out',
163 url: this.state.image
164 })
165 }
166
167 _copyToClipboard = () => {
168 Clipboard.setString(this.state.image)
169 alert('Copied to clipboard')
170 }
171
172 _takePhoto = async () => {
173 let pickerResult = await ImagePicker.launchCameraAsync({
174 allowsEditing: true,
175 aspect: [4, 3]
176 })
177
178 this._handleImagePicked(pickerResult)
179 }
180
181 _pickImage = async () => {
182 let pickerResult = await ImagePicker.launchImageLibraryAsync({
183 allowsEditing: true,
184 aspect: [4, 3]
185 })
186
187 this._handleImagePicked(pickerResult)
188 }
189
190 _handleImagePicked = async pickerResult => {
191 try {
192 this.setState({ uploading: true })
193
194 if (!pickerResult.cancelled) {
195 uploadUrl = await uploadImageAsync(pickerResult.uri)
196 this.setState({ image: uploadUrl })
197 }
198 } catch (e) {
199 console.log(e)
200 alert('Upload failed, sorry :(')
201 } finally {
202 this.setState({ uploading: false })
203 }
204 }
205
206 submitToGoogle = async () => {
207 try {
208 this.setState({ uploading: true })
209 let { image } = this.state
210 let body = JSON.stringify({
211 requests: [
212 {
213 features: [
214 { type: 'LABEL_DETECTION', maxResults: 10 },
215 { type: 'LANDMARK_DETECTION', maxResults: 5 },
216 { type: 'FACE_DETECTION', maxResults: 5 },
217 { type: 'LOGO_DETECTION', maxResults: 5 },
218 { type: 'TEXT_DETECTION', maxResults: 5 },
219 { type: 'DOCUMENT_TEXT_DETECTION', maxResults: 5 },
220 { type: 'SAFE_SEARCH_DETECTION', maxResults: 5 },
221 { type: 'IMAGE_PROPERTIES', maxResults: 5 },
222 { type: 'CROP_HINTS', maxResults: 5 },
223 { type: 'WEB_DETECTION', maxResults: 5 }
224 ],
225 image: {
226 source: {
227 imageUri: image
228 }
229 }
230 }
231 ]
232 })
233 let response = await fetch(
234 'https://vision.googleapis.com/v1/images:annotate?key=' +
235 Environment['GOOGLE_CLOUD_VISION_API_KEY'],
236 {
237 headers: {
238 Accept: 'application/json',
239 'Content-Type': 'application/json'
240 },
241 method: 'POST',
242 body: body
243 }
244 )
245 let responseJson = await response.json()
246 console.log(responseJson)
247 this.setState({
248 googleResponse: responseJson,
249 uploading: false
250 })
251 } catch (error) {
252 console.log(error)
253 }
254 }
255}
256
257async function uploadImageAsync(uri) {
258 const blob = await new Promise((resolve, reject) => {
259 const xhr = new XMLHttpRequest()
260 xhr.onload = function() {
261 resolve(xhr.response)
262 }
263 xhr.onerror = function(e) {
264 console.log(e)
265 reject(new TypeError('Network request failed'))
266 }
267 xhr.responseType = 'blob'
268 xhr.open('GET', uri, true)
269 xhr.send(null)
270 })
271
272 const ref = firebase
273 .storage()
274 .ref()
275 .child(uuid.v4())
276 const snapshot = await ref.put(blob)
277
278 blob.close()
279
280 return await snapshot.ref.getDownloadURL()
281}
282
283const styles = StyleSheet.create({
284 container: {
285 flex: 1,
286 backgroundColor: '#fff',
287 paddingBottom: 10
288 },
289 developmentModeText: {
290 marginBottom: 20,
291 color: 'rgba(0,0,0,0.4)',
292 fontSize: 14,
293 lineHeight: 19,
294 textAlign: 'center'
295 },
296 contentContainer: {
297 paddingTop: 30
298 },
299
300 getStartedContainer: {
301 alignItems: 'center',
302 marginHorizontal: 50
303 },
304
305 getStartedText: {
306 fontSize: 17,
307 color: 'rgba(96,100,109, 1)',
308 lineHeight: 24,
309 textAlign: 'center'
310 },
311
312 helpContainer: {
313 marginTop: 15,
314 alignItems: 'center'
315 }
316})

Note that, most of the source code for accessing and uploading to Firebase is taken from an example of using Expo with Firebase here. I am going to explain below the bits that are essential to connect and run Firebase. First, let us start by understanding what uploadImageAsync is doing.

1async function uploadImageAsync(uri) {
2 const blob = await new Promise((resolve, reject) => {
3 const xhr = new XMLHttpRequest()
4 xhr.onload = function() {
5 resolve(xhr.response)
6 }
7 xhr.onerror = function(e) {
8 console.log(e)
9 reject(new TypeError('Network request failed'))
10 }
11 xhr.responseType = 'blob'
12 xhr.open('GET', uri, true)
13 xhr.send(null)
14 })
15
16 const ref = firebase
17 .storage()
18 .ref()
19 .child(uuid.v4())
20 const snapshot = await ref.put(blob)
21
22 blob.close()
23
24 return await snapshot.ref.getDownloadURL()
25}

As shown in the above snippet, the uploadImageAsync function uploads the image by creating a unique image ID or blob with the help of uuid. It also uses xhr to send a request to the Firebase storage to upload the image. We are also defining a default state in the App component and asking for User Permissions for both using the camera roll or gallery or take a photo from the device's camera as shown in the code snippet below.

1state = {
2 image: null,
3 uploading: false,
4 googleResponse: null
5 };
6
7 async componentDidMount() {
8 await Permissions.askAsync(Permissions.CAMERA_ROLL);
9 await Permissions.askAsync(Permissions.CAMERA);
10 }

The Button in our App component publishes the image to Google's Cloud Vision API.

1<Button
2 style={{ marginBottom: 10 }}
3 onPress={() => this.submitToGoogle()}
4 title="Analyze!"
5/>

The submitToGoogle method is what sends requests and communicates with the API to fetch the result when the button Analyze is pressed by the user.

1submitToGoogle = async () => {
2 try {
3 this.setState({ uploading: true });
4 let { image } = this.state;
5 let body = JSON.stringify({
6 requests: [
7 {
8 features: [
9 { type: "LABEL_DETECTION", maxResults: 10 },
10 { type: "LANDMARK_DETECTION", maxResults: 5 },
11 { type: "FACE_DETECTION", maxResults: 5 },
12 { type: "LOGO_DETECTION", maxResults: 5 },
13 { type: "TEXT_DETECTION", maxResults: 5 },
14 { type: "DOCUMENT_TEXT_DETECTION", maxResults: 5 },
15 { type: "SAFE_SEARCH_DETECTION", maxResults: 5 },
16 { type: "IMAGE_PROPERTIES", maxResults: 5 },
17 { type: "CROP_HINTS", maxResults: 5 },
18 { type: "WEB_DETECTION", maxResults: 5 }
19 ],
20 image: {
21 source: {
22 imageUri: image
23 }
24 }
25 }
26 ]
27 });
28 let response = await fetch(
29 "https://vision.googleapis.com/v1/images:annotate?key=" +
30 Environment["GOOGLE_CLOUD_VISION_API_KEY"],
31 {
32 headers: {
33 Accept: "application/json",
34 "Content-Type": "application/json"
35 },
36 method: "POST",
37 body: body
38 }
39 );
40 let responseJson = await response.json();
41 console.log(responseJson);
42 this.setState({
43 googleResponse: responseJson,
44 uploading: false
45 });
46 } catch (error) {
47 console.log(error);
48 }
49 };
50}

The Vision API uses HTTP Post request as a REST API endpoint to perform data analysis on images you send in the request. This is done via the URL https://vision.googleapis.com/v1/images:annotate. To authenticate each request, we need the API key. The body of this POST request is in JSON format. For example:

1{
2 "requests": [
3 {
4 "image": {
5 "content": "/9j/7QBEUGhvdG9...image contents...eYxxxzj/Coa6Bax//Z"
6 },
7 "features": [
8 {
9 "type": "LABEL_DETECTION",
10 "maxResults": 1
11 }
12 ]
13 }
14 ]
15}

You can change the value of maxResults for every category. The response from the Vision API is also in JSON format.

1"labelAnnotations": Array [
2 Object {
3 "description": "water",
4 "mid": "/m/0838f",
5 "score": 0.97380537,
6 "topicality": 0.97380537,
7 },
8 Object {
9 "description": "waterfall",
10 "mid": "/m/0j2kx",
11 "score": 0.97099465,
12 "topicality": 0.97099465,
13 },
14 Object {
15 "description": "nature",
16 "mid": "/m/05h0n",
17 "score": 0.9594912,
18 "topicality": 0.9594912,
19 }
20]

The above result can be viewed in the terminal from Expo logs. You can see how the application works with a short demo done on iOS simulator below.

demo

If you visit the storage section in Firebase, you can notice that each image is stored with a name of base64 binary string.

ss5

If you have a real device, just download the Expo client, scan the QR code and then you can try the Take a photo feature inside the application.

Conclusion

In this tutorial, we’ve shown you how to integrate Firebase storage services and use a machine learning API such as Google's Vision API with a React Native and Expo application.

You can find the complete code inside this Github repo.

Originally published at Jscrambler

I'm Aman working as an independent fullstack developer with technologies such as Node.js, ReactJS, and React Native. I try to document and write tutorials to help JavaScript, Web and Mobile developers.