Build an ML Rock Paper Scissors Game with Lobe

by Adi Azulay and

published March 31, 2021, last updated March 31, 2021

posted in Adafruit Products Sensors/ Camera Raspberry Pi Programming/ Python Machine Learning

Save Download

19

Updated

Intermediate

Project guide

Overview

Classify hand gestures to play Rock, Paper, Scissors on your Pi using computer vision and machine learning! Lobe is a free, easy to use app that lets you train custom machine learning (ML) models without any code. Download Lobe here.

In this tutorial, we'll build a Rock, Paper, Scissors ML model and play against our Pi. The game will classify your hand gesture using the Pi camera, let the computer randomly pick one, and determine who the winner is.

This is a fun introduction to building interactive ML models with Lobe on the Pi. Use this as a starting point to explore your own ideas!

This tutorial is part of a series which includes:

Background Knowledge

New to using Lobe? Check out our introductory tutorial:

Tutorial 1: Machine Learning 101 with Lobe and Raspberry Pi

We also recommend that you have familiarity with the following topics:

Setting up and using the Raspberry Pi
Some experience terminal window
Some experience with Python

Text editor powered by tinymce.

Required Parts

Angled shot of assembled BrainCraft HAT with Raspberry Pi Camera and long FPC ribbon cable.

Microsoft Machine Learning Kit for Lobe with Raspberry Pi 4 4GB

Machine learning is a transformative tool that’s redefining how we build software—but up until now, it was only accessible to a small group of experts. At Adafruit, we...

guides with product

$139.95

In Stock

Add to Cart

Text editor powered by tinymce.

Set up your Pi

Hardware Setup

Before starting, set up your Pi and BrainCraft HAT. Follow this primary guide for the BrainCraft to configure these services:

Blinka
Fan Service
Display Module

Installing Lobe SDK

Connect to your Pi using SSH and run the following script to install the Lobe SDK:

Copy Code

cd ~
wget https://raw.githubusercontent.com/lobe/lobe-python/master/scripts/lobe-rpi-install.sh
sudo bash lobe-rpi-install.sh

cd ~
wget https://raw.githubusercontent.com/lobe/lobe-python/master/scripts/lobe-rpi-install.sh
sudo bash lobe-rpi-install.sh

Note: To setup SSH, follow this guide.

Text editor powered by tinymce.

Train a Gesture Detector

The first part of our Rock, Paper, Scissors game is to train an ML model that can classify the hand gesture you make.

We'll use Lobe to collect images to create a dataset and train the model.

Collect your images in Lobe

First, download and install Lobe from the link below:

camera_1._Lobe_New_Project.png

Open Lobe and create a new project.

camera_1._empty_project.png

From the top right, select Import and choose Camera in the drop-down menu.

camera_3._Rock.png

In the bottom left corner, set your first label to "Rock" and take at least 10 pictures of your fist at different angles.

Hint: If you press and hold the capture button you can take a burst of images in Lobe.

Make sure your model labels are capitalized to work with the sample code.

camera_4._Paper.png

camera_6._All_images.png

Repeat this step for the "Paper" and "Scissors" hand gestures.

Test your model

In Lobe, switch to the Use tab in the left menu and select Camera on the top.

camera_8._Test_Scissors.png

Test all three hand gestures and check that the model is recognizing them accurately. If the model is not working as expected, you can use the green and red buttons to improve the model.

Click the green button when your model predicts the correct label. This will add the image with the correct label to your dataset.

Click the red button when your model predicts the wrong label. You can then provide the correct label and the image will get added to your dataset.

If you find that one of the gestures is consistently confusing the model, try collecting more images of that gesture.

Text editor powered by tinymce.

Export your Model

Next, export your Lobe model to use on the Raspberry Pi. We'll use TensorFlow Lite which is a format that is optimized for mobile and edge devices, like the Pi.

adafruit_products_6._Export.png

In Lobe, navigate to the Use tab and click Export.

Select TensorFlow Lite and select a location to save the model. We'll transfer the model to our Raspberry Pi later in the tutorial.

Text editor powered by tinymce.

Set up an FTP Connection

Since we're using the Pi in a headless configuration, we'll use an FTP connection to transfer files between our computer and the Pi.

Windows Instructions

Download and install WinSCP

Open WinSCP and start a New Session

Select an SFTP connection, fill in the IP address of your Pi, set the username to Pi, and put in your password.

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Mac Instructions

Download and install FileZilla. When it's done installing, open the program.

Type sftp:// followed by the IP address of your Pi. Set the username to pi and put in your password.

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Text editor powered by tinymce.

Download the Code

Download the GitHub repo into the Pi's home folder with the following commands:

Copy Code

cd ~
git clone https://github.com/lobe/lobe-adafruit-kit.git

cd ~
git clone https://github.com/lobe/lobe-adafruit-kit.git

The GitHub repo can also be found here.

Text editor powered by tinymce.

Test your Model on the Pi

Transfer the model to the Pi

Connect your Pi to a power source.

Open command prompt on a PC or terminal on Mac/Linux. Connect to your Pi using SSH.

The command will look like the following, replacing the numbers after the @ sign with the IP address of your Pi:

Copy Code

ssh [email protected]

ssh [email protected]

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Create a new folder in the home directory and name it model by using the following commands:

Copy Code

cd ~
mkdir model

cd ~
mkdir model

Open the FTP connection and use it to copy saved_model.tflite and signature.json from your exported Lobe model into the model directory on the Pi.

Follow the same procedure for FileZilla.

Test your model

Use the commands below to download and run the basic Lobe classifier on your Pi:

Copy Code

cd ~
cd lobe-adafruit-kit
python3 lobe-basic-prediction.py

cd ~
cd lobe-adafruit-kit
python3 lobe-basic-prediction.py

Make sure that all the Rock, Paper, Scissors hand gestures are working as expected. It's best to keep the camera background consistent with the images you used for training.

Text editor powered by tinymce.

Play Rock, Paper, Scissors

Now that your model is trained and tested we can integrate it with a game of Rock, Paper, Scissors! The game demonstrates how to use the predicted label from the model to trigger events.

First, navigate to the Lobe Adafruit Kit folder:

Copy Code

cd ~
cd lobe-adafruit-kit

cd ~
cd lobe-adafruit-kit

Run the Python program with the following command:

Copy Code

python3 lobe-rock-paper-scissors.py

python3 lobe-rock-paper-scissors.py

Play the Game!

Click the BrainCraft button. This starts a countdown -- make your hand gesture and the Pi will take a photo at the end of the countdown.

The screen displays the Pi's choice and then displays a winner!

Text editor powered by tinymce.

Game Code Overview

This page dives deeper into how the Rock, Paper, Scissors game code works.

Copy Code

signs = ['Rock', 'Paper', 'Scissors']

game_logic = {'Rock' : 'Paper',
	      'Paper' : 'Scissors',
	      'Scissors' : 'Rock'}

signs = ['Rock', 'Paper', 'Scissors']

game_logic = {'Rock' : 'Paper',
	      'Paper' : 'Scissors',
	      'Scissors' : 'Rock'}

First, we define the three hand gestures that the player and computer can play.

Next, we define the game logic using a dictionary. The dictionary value on the right beats the dictionary key on the left. For example, the dictionary key 'Rock' is beaten by the dictionary value 'Paper'.

Copy Code

def load_image(path, camera) -> Image:
	img = Image.open(path)

	pad = Image.new('RGB', (
		((img.size[0] + 31) // 32) * 32,
		((img.size[1] + 15) // 16) * 16,
		))

	pad.paste(img, (0, 0))
	
	layer = camera.add_overlay(pad.tobytes(), size=img.size)

	return layer

def load_image(path, camera) -> Image:
	img = Image.open(path)

	pad = Image.new('RGB', (
		((img.size[0] + 31) // 32) * 32,
		((img.size[1] + 15) // 16) * 16,
		))

	pad.paste(img, (0, 0))
	
	layer = camera.add_overlay(pad.tobytes(), size=img.size)

	return layer

Next, we include a function to load images as layers. We use these images for the countdown and to show the hand signs that the computer chooses. The function scales the images to proper size to fit on the screen.

Copy Code

def random_sign():
	return random.choice(signs)

def compare_signs(player_sign, computer_sign):
	if (game_logic[player_sign] == computer_sign):
		print('computer wins')
		return 
	elif (game_logic[computer_sign] == player_sign):
		print('you win')
		return
	else:
		print('tie')
		return

def random_sign():
	return random.choice(signs)

def compare_signs(player_sign, computer_sign):
	if (game_logic[player_sign] == computer_sign):
		print('computer wins')
		return 
	elif (game_logic[computer_sign] == player_sign):
		print('you win')
		return
	else:
		print('tie')
		return

The first function above uses random.choice to return a random element from the list of signs.

The second function compares the sign detected by the Lobe model with the sign the Raspberry Pi chose randomly. It checks whether the sign the player chose loses to the sign the computer chose.

For example, if the player chooses Rock, then game_logic[player_sign] will equal Paper since we're getting the element that is at game_logic index Rock.

Main program code

Copy Code

model = ImageModel.load('~/model')

model = ImageModel.load('~/model')

ImageModel is a class from the Lobe library. We load the Lobe model and create an instance of the class.

Copy Code

with picamera.PiCamera(resolution=(224, 224), framerate=30) as camera:

with picamera.PiCamera(resolution=(224, 224), framerate=30) as camera:

Next, we instantiate the Pi Camera. The rest of the code runs inside this indent block.

Copy Code

stream = io.BytesIO()
camera.start_preview()

time.sleep(2)

stream = io.BytesIO()
camera.start_preview()

time.sleep(2)

We create a stream to continuously show the camera footage, then start the preview on the camera to populate the stream, and wait 2 seconds to let the camera warm up.

Copy Code

rock = load_image('assets/rock.png', camera)
paper = load_image('assets/paper.png', camera)
scissor = load_image('assets/scissor.png', camera)
counter_one = load_image('assets/one.png', camera)
counter_two = load_image('assets/two.png', camera)
counter_three = load_image('assets/three.png', camera)

rock = load_image('assets/rock.png', camera)
paper = load_image('assets/paper.png', camera)
scissor = load_image('assets/scissor.png', camera)
counter_one = load_image('assets/one.png', camera)
counter_two = load_image('assets/two.png', camera)
counter_three = load_image('assets/three.png', camera)

The above code loads all the images we use in the game. Each image is loaded into a separate layer.

Main loop

The section below covers the main program loop.

Copy Code

stream.seek(0)

stream.seek(0)

The above code returns the stream to the first byte.

Copy Code

inputs = get_inputs()
while Input.BUTTON not in inputs:
	inputs = get_inputs()
	time.sleep(0.1)

inputs = get_inputs()
while Input.BUTTON not in inputs:
	inputs = get_inputs()
	time.sleep(0.1)

Next, we wait for the button to be pressed before starting the game.

Copy Code

camera.preview.alpha = 0
counter_one.layer = 3
time.sleep(1)

counter_one.layer = 0
counter_two.layer = 3
time.sleep(1)

counter_two.layer = 0
counter_three.layer = 3
time.sleep(1)

counter_three.layer = 0
camera.preview.alpha = 255
time.sleep(1)

camera.preview.alpha = 0
counter_one.layer = 3
time.sleep(1)

counter_one.layer = 0
counter_two.layer = 3
time.sleep(1)

counter_two.layer = 0
counter_three.layer = 3
time.sleep(1)

counter_three.layer = 0
camera.preview.alpha = 255
time.sleep(1)

Once the button is pressed, we turn the opacity of the camera preview to 0 so it's transparent. Then we move each number of the countdown to the front and wait 1 second between each one. Finally we turn the camera opacity back to 255 (opaque).

Copy Code

camera.capture(stream, format='jpeg')
img = Image.open(stream)

result = model.predict(img)

label = result.prediction

camera.annotate_text = label
time.sleep(0.5)

camera.capture(stream, format='jpeg')
img = Image.open(stream)

result = model.predict(img)

label = result.prediction

camera.annotate_text = label
time.sleep(0.5)

The next step is to capture an image from the camera stream and open it as a Pillow image. The Lobe model uses inferencing to make a prediction on the image, and returns the predicted label. This is added to the camera preview.

Copy Code

computer_sign = random_sign()

if (computer_sign == 'Rock'):
	Rock.layer = 3
elif (computer_sign == 'Paper'):
	Paper.layer = 3
elif (computer_sign == 'Scissors'):
	Scissors.layer = 3

time.sleep(2)

Rock.layer = 0
Paper.layer = 0
Scissors.layer = 0

computer_sign = random_sign()

if (computer_sign == 'Rock'):
	Rock.layer = 3
elif (computer_sign == 'Paper'):
	Paper.layer = 3
elif (computer_sign == 'Scissors'):
	Scissors.layer = 3

time.sleep(2)

Rock.layer = 0
Paper.layer = 0
Scissors.layer = 0

The above code is how the Pi plays the game! The Pi generates a random sign, checks which sign was generated, and shows the corresponding image for 2 seconds.

Copy Code

winner = compare_signs(label, computer_sign)

if (winner == 'player'):
	camera.annotate_text = 'You Win!'
elif (winner == 'computer'):
	camera.annotate_text = 'You Lose...'
elif (winner == 'tie'):
	camera.annotate_text = 'Tie'

winner = compare_signs(label, computer_sign)

if (winner == 'player'):
	camera.annotate_text = 'You Win!'
elif (winner == 'computer'):
	camera.annotate_text = 'You Lose...'
elif (winner == 'tie'):
	camera.annotate_text = 'Tie'

Finally, we compare the player gesture (label from the prediction) and the computer gesture, check who the winner was and add the text to the camera preview.

Text editor powered by tinymce.

Going Forward

Now that you've built a Rock, Paper, Scissors game for yourself, try making it more accessible for other folks!

The model is only as good as the data you give it. To make this model work for more people, collect images of many different hands. If you can train with hands that are feminine and masculine, old and young, small and large, and with different skin tones, you'll end up with a better model that will work for more people.

You can also use this project as a template to do other types of gesture applications and trigger different actions with the BrainCraft.

To learn how to update your Lobe model with images taken from the Pi Camera, check out this tutorial:

Tutorial 3: Build an ML Package Detector with Lobe

Happy making!

Text editor powered by tinymce.

Search