WWDC 2017 brought many amazing and powerful APIs to iOS. Among the most popular isCore ML(of course, ARKit is hot too!). Core ML allows developers to use machine learning models in their apps without the knowledge of neural networks or machine learning algorithms. Today, I will show you how simple it is to create a game using Core ML. We will be creating a simple scavenger hunt game which will have the player running around the room looking for objects. We have already coveredthe basics of Core ML in our tutorialwhich I suggest you read over before continuing.
Note:You will need Xcode 9 beta to for this tutorial. You will also need a device that is running iOS 11 beta in order to test out the app. This app will NOT run in the simulator. The also might run slow on older devices such as the iPhone 5s. While Xcode 9 beta supports both Swift 3.2 and 4.0, all the code in this tutorial is written in Swift 4.
The game that we are making today is simple to create and play. After the user taps the Start button, the name of a random object will appear at the top of the screen. It is the player’s job to find that object. Once the player does, they will point the phone at it. Using the machine learning algorithm, the iPhone will recognize the object and then ask for another one. Every object found adds a point to the score. The player also has the option to skip an object if they cannot find it.
This app recognizes objects slightly differently than the one demonstrated in the Introduction to Core ML tutorial. This is mainly because we will have a live camera view and predicting objects in real time rather than choosing a single photo.
First things first! Let’s open Xcode 9 beta and create a new project. Choose the_Single View App_template for this app. Though we are building a game, the Single View Application template is good enough. I will be naming the project_CoreMLScavenge_but you can name it whatever you wish. Make sure to set the language toSwift.
Once the project is created, uncheck the boxes next to
Landscape Right. We will stick to portrait mode for this game.
Now it’s time to have fun! Go to the
Main.storyboardfile in the project navigator. Start by adding a
Viewto the top and bottom of the View Controller. Stretch the widths to the size of the View Controller and make the height around 85 pixels for each. The entire background is going to be the live camera feed, so we are adding these views so we can see the labels and buttons on top of it. Make the background of the view a light-gray color. This way we know not to accidentally put any elements on top of it.
In the top view, add a
UILabelto the center and another one to the left. The center label will show the object for the player to find, so stretch it out a bit so it can fit longer words. The label on the left will show the score. You can condense it a bit since it will only need to show two digits.
In the bottom view, add two
UILabels and two
UIButtons. One label will show the score, and the other will show the player’s highest score. Make one of the buttons say “start”, which will of course start the game. Make the other button say “skip”. The players can tap this if they cannot find the object and want another one.
I will not cover how to add layout constraints to these elements in this tutorial. However, I highly suggest that you do. Check out ourAuto Layout Tutorial if you would like to learn how to use it. If you are not going to use Auto Layout, be sure to make your view controller’s size to the type of the device you are going to run the app on.
Now that the UI is done, we can start coding. Select the
ViewController.swiftfile in the project navigator. Let’s start by importing the necessary frameworks above the class.
Next add the outlets so we can connect our code to the UI. We need to add 8 outlets in total.
Your code should now look like this:
Now we need to create some global variables. Let’s add the following variables right under where we added the outlets.
Let me explain each variable one by one:
Line #1:This is the camera layer which we will add to the view later on. It will take up the entire background of the screen.
Line #2:We will initialize the game timer later. We are making it global so we can invalidate it (stop it) from any function.
Line #3:This variable will hold the time remaining in the game and it is initialized to 60 so the game lasts one minute. Feel free to change it to another number if you want the game to be shorter or longer.
Line #4:This will hold the player’s score and increase by one each time they find an object.
Line #5:This will hold the player’s high score. It will be set from UserDefaults each time the app loads.
Putting a lot of code in the
viewDidLoadcan make our code look messy and unorganized. To fix this, we’re going to add a
viewSetupfunction right after the
viewDidLoadto take care of some of the basic UI settings that need to be done at runtime.
Add this into the
Then add this right after the
What this is doing is adding some transparency to the top and bottom views so we can see the live camera feed behind them. We are also setting the score label text to 0.
Your code should now look like this:
Now it’s time to setup the live camera feed which will the entire background of the view. Before we start adding code, we need to ask the user’s permission to use the camera. iOS will take care of most of the work related to this. However, we need to provide a description of why we need the camera.
Head over to the
info.plistfile in the project navigator. Add a row with the key_Privacy – Camera Usage Description_and type a description in the value.
Now it’s time to add code. To keep things organized, we will add the code to prepare the camera in a function called
Add the following to the
Next, add the following function after the
Now we are going to create something called
AVCaptureSessionwhich will make it possible to have a real time capture. Add the following to the
So what is going on here?
Line #1:We create the
Line #2:We set the preset for the caption which pretty much sets the quality for the output. We are setting it to photo for it to be high resolution.
Line #3:We create an AVCaptureDevice which is the back camera. There is no reason to use the front camera for this game.
Line #4:We specify that we are using the back camera by referencing the backCamera AVCaptureDevice.
Line #5:We are adding the backCamera device as the input for the captureSession.
Remember the variable we made earlier called
cameraLayer? Now it’s time to use it. We’re going to be adding the camera layer as a sublayer and setting it to the view controller’s size.
Add the following after the code we just added for the capture session:
Line #1-3:Here we are initializing the
AVCaptureVideoPreviewLayerand setting the setting parameter to
captureSession. After that, we add it to the view as a sublayer and set the size to the size of the entire view controller.
Line #5-6:Here we are bringing the top and bottom views to the front. This way, the camera layer doesn’t cover them.
To finish off this function, add the following code under what we just added:
Line #1-3:Here we are creating the data output and specifying the output settings. You will see an error here but don’t worry about it. We will take care of that soon.
Line #5-7:Finally, we add the video output to the capture session and start running it.
cameraSetupfunction should now look like this:
After setting up the UI and camera, it is time to work with Core ML for object recognition.
Before we continue coding, we need to add the Core ML data model into the project. In order to use Core ML, you need to have a pre-trained model. While it is possible to build your own model for this game, we will use a model available on Apple’s developer website.
Go to“Apple’s machine learning website”and scroll to the bottom. Here you will find four different pre-trained Core ML models available for download.
For this game, we will be using the_Inception V3_model. Once you have the model downloaded, drag it into the Xcode project’s project navigator. Click on it and observe what is displayed.
Note:Please make sure that Target Membership of the project is selected, otherwise, your app will not be able to access the file.
As you can see, this model has a type of aNeural Network Classifier. It takes in an image size 299×299 and outputs two things. A dictionary which is the probability or confidence level for the calculations of the model, and a string with the actual name of the object in the picture.
Now our little field trip is over and it’s time go to back to the
View Controllerand add two awesome functions that make the magic happen.
First we’re going to add the
predictfunction. Add the following code after the
Don’t let its size fool you, this is a powerful function. Here is what the code does:
Line #2:We create a model constant for the Inception V3 model.
Line #3:We create a
VNCoreMLRequestwhich will call the results function (we are writing that next, so don’t worry about the error.)
Line #4-5:We create a constant for a
VNSequenceRequestHandlerobject and then perform it with the image passed into the predict function via its parameter.
Next, it’s time to add the
resultsfunction which will handle the results of the
predictmethod and keep the game moving forward.
Add the following code after the
This function is called after a prediction. It first has two guards to make sure that there is actually a result. If there is, it will take the result with the highest confidence and put it into its own string called
identifier. Then the function will check to see if the identifier is the same as the object label’s text. If they are equal, that means the player found the correct object and the score will be increased. There is also a function call for
nextObjectwhich is commented out. We have not yet created that function so leave it that way for now.
Okay! So we are done with the majority of the Core ML code for the game. Isn’t that easy? Your
resultsfunctions should look like this:
One more thing we need to do before we are done with the Core ML part of this game. We need to add an extension to call the predict function. Add the following code the the very bottom of the file, after the last bracket:
This extension adopts the
AVCaptureVideoDataOutputSampleBufferDelegateprotocol to process the captured video sample buffers, and creates a CGImage from the video feed. With the images, we pass it to the predict function. You should now be able to build your project without any errors. This is how we continuously capture video and pass it to the built-in machine learning model for object recognition.
In order for the game to give the player random objects to look for, we have to prepare a list of objects to choose from. To do this, we will create an array with the names of a bunch of objects. Later we will create a function to choose an element from this array randomly and tell the player to go find it.
This is going to be a somewhat lengthy array, so for organization purposes, I have decided to create it in a struct in a separate Swift file. Right click in the project navigator and choose
Now select the Swift File option and clicknext. Name the file “Objects” and clickcreate. Place the following code snippet into the file after the
Here I have created an array called
objectArrayof some common, random, household objects that the Core ML model looks for. These are just a select few so feel free to add more.
Now go back to
ViewController.swiftbefore we jump into the next step.
We will be saving the player’s highest score using
UserDefaults. This means that we need a setter function to save the score into UserDefaults and a getter function to retrieve it.
Add the following two functions to your code after the results function:
The first function,
getHighScore, uses an
if letstatement to check if there is already a saved score in UserDefaults. If there is, it sets the
highScorevariable to it, if not, it sets the
highScoreto 0. The second function,
setHighScore, simply sets the current score as the high score. We will call this when the player beats their score.
getHighScorefunction needs to be called when the app loads up. This way, the player can see their high score when they open the app.
cameraSetupfunction call. Your
ViewDidLoadshould look like this:
Now it’s time to add a number of different functions to pull everything we have already done together and make this app work. We will start by adding functions to choose a random object from the objects array and handle when the game ends.
Add the following to your code after the
So what exactly are these functions doing you ask?
function is called when the timer is up.
- We first unhide the start button and hide the skip button since it is no longer needed when the game is not in play. We also set the object label’s text to say “Game Over”.
- We check if the score that the player just got is higher than the high score. If it is, than we call the
- We reset the variables for the next game.
function is called when the player either finds the correct object or taps the skip button. This function will take a random object from the object array and set it to the objects label so the player knows to find it.
- We create an instance of the objectArray.
- We generate a random number from 0 to the length of the object array.
- We use a guard to make sure that the random number that was just chosen is not the same as the last one. This way, the player will not get the same object twice in a row.
- We set the object label to the random object and make sure the score label is displaying the correct score.
Important:Now that we have created the_nextObject_function, make sure you go back up to the_results_function and uncomment the function call fornextObject.
Next, we need to add the actions that we will connect to the play and skip buttons. Add the following code after the
So what did we do here?
- First we added an action that will be called when the start button is tapped. Here we initialize the gameTimer using the new method that takes a block to invoke rather than a selector which makes the code much cleaner and more organized.
- We add a guard to make sure that there is time left in the game. If there is not, we invalidate the timer and run the
function. If there is time, we subtract one from the time remaining and update the timer label.
- Here we hide the startButton, show the skip button, and call the
function to show the first object to find.
- We add an action to connect to the skip button which simply calls for a new object.
Important:Now, make sure you go to your storyboard and connect all outlets an actions to the proper elements.
Now it’s finally time to build and run your game. Once loaded, tap the play button and point your phone at the object stated at the top. See how high is your score! You might need to wait a second for the app to focus before it can recognize an object. Also, please note that if you have an older device such as an iPhone 5s or 6, the app might run really slow because of the power and resources that Core ML needs. As I testedt, the game ran great on my iPhone 7. But when I tried it on my iPhone 5s, it was really slow and laggy.
In case you were wondering, this model cannot recognize the Nintendo Switch!
I hope you had fun creating this game and have some more insight on how much you can do with Core ML. The game is not perfect, there are a lot of things you can improve (say, play a sound when the object is matched). But this is just a simple example showing how you can apply Core ML. So get creative and see how you can improve upon it.
For reference, the complete demo app is available onGitHub.
For more details about the Core ML framework, you can refer to theofficial Core ML documentation.
Do you like this tutorial? Feel free to leave a comment and let me know.