Classify hand gestures to play Rock, Paper, Scissors on your Pi using computer vision and machine learning! Lobe is a free, easy to use app that lets you train custom machine learning (ML) models without any code. Download Lobe here.

In this tutorial, we'll build a Rock, Paper, Scissors ML model and play against our Pi. The game will classify your hand gesture using the Pi camera, let the computer randomly pick one, and determine who the winner is.

This is a fun introduction to building interactive ML models with Lobe on the Pi. Use this as a starting point to explore your own ideas!

This tutorial is part of a series which includes:

Background Knowledge

New to using Lobe? Check out our introductory tutorial:

We also recommend that you have familiarity with the following topics:

  1. Setting up and using the Raspberry Pi
  2. Some experience terminal window
  3. Some experience with Python
Microsoft Machine Learning Kit for Lobe with Raspberry Pi 4 4GB
Machine learning is a transformative tool that’s redefining how we build software—but up until now, it was only accessible to a small group of experts. At Adafruit, we...
Out of Stock

Hardware Setup

Before starting, set up your Pi and BrainCraft HAT. Follow this primary guide for the BrainCraft to configure these services:

  • Blinka
  • Fan Service
  • Display Module

Installing Lobe SDK

Connect to your Pi using SSH and run the following script to install the Lobe SDK:

cd ~
sudo bash

Note: To setup SSH, follow this guide.

The first part of our Rock, Paper, Scissors game is to train an ML model that can classify the hand gesture you make.

We'll use Lobe to collect images to create a dataset and train the model.

Collect your images in Lobe

First, download and install Lobe from the link below:

Open Lobe and create a new project.

From the top right, select Import and choose Camera in the drop-down menu.

In the bottom left corner, set your first label to "Rock" and take at least 10 pictures of your fist at different angles.

Hint: If you press and hold the capture button you can take a burst of images in Lobe.

Make sure your model labels are capitalized to work with the sample code.

Repeat this step for the "Paper" and "Scissors" hand gestures.

Test your model

In Lobe, switch to the Use tab in the left menu and select Camera on the top.

Test all three hand gestures and check that the model is recognizing them accurately. If the model is not working as expected, you can use the green and red buttons to improve the model.

Click the green button when your model predicts the correct label. This will add the image with the correct label to your dataset.

Click the red button when your model predicts the wrong label. You can then provide the correct label and the image will get added to your dataset. 

If you find that one of the gestures is consistently confusing the model, try collecting more images of that gesture.

Next, export your Lobe model to use on the Raspberry Pi. We'll use TensorFlow Lite which is a format that is optimized for mobile and edge devices, like the Pi. 

In Lobe, navigate to the Use tab and click Export.

Select TensorFlow Lite and select a location to save the model. We'll transfer the model to our Raspberry Pi later in the tutorial.

Since we're using the Pi in a headless configuration, we'll use an FTP connection to transfer files between our computer and the Pi.

Windows Instructions

Download and install WinSCP


Open WinSCP and start a New Session

Select an SFTP connection, fill in the IP address of your Pi, set the username to Pi, and put in your password.

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Mac Instructions

Download and install FileZilla. When it's done installing, open the program.

Type sftp:// followed by the IP address of your Pi. Set the username to pi and put in your password.

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Download the GitHub repo into the Pi's home folder with the following commands:

cd ~
git clone

The GitHub repo can also be found here.

Transfer the model to the Pi

Connect your Pi to a power source.

Open command prompt on a PC or terminal on Mac/Linux. Connect to your Pi using SSH.

The command will look like the following, replacing the numbers after the @ sign with the IP address of your Pi:

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Create a new folder in the home directory and name it model by using the following commands:

cd ~
mkdir model

Open the FTP connection and use it to copy saved_model.tflite and signature.json from your exported Lobe model into the model directory on the Pi.

Follow the same procedure for FileZilla.

Test your model

Use the commands below to download and run the basic Lobe classifier on your Pi:

cd ~
cd lobe-adafruit-kit

Make sure that all the Rock, Paper, Scissors hand gestures are working as expected. It's best to keep the camera background consistent with the images you used for training.

Now that your model is trained and tested we can integrate it with a game of Rock, Paper, Scissors! The game demonstrates how to use the predicted label from the model to trigger events.

First, navigate to the Lobe Adafruit Kit folder:

cd ~
cd lobe-adafruit-kit

Run the Python program with the following command:


Play the Game!

Click the BrainCraft button. This starts a countdown -- make your hand gesture and the Pi will take a photo at the end of the countdown.

The screen displays the Pi's choice and then displays a winner!

This page dives deeper into how the Rock, Paper, Scissors game code works.

signs = ['Rock', 'Paper', 'Scissors']

game_logic = {'Rock' : 'Paper',
	      'Paper' : 'Scissors',
	      'Scissors' : 'Rock'}

First, we define the three hand gestures that the player and computer can play.

Next, we define the game logic using a dictionary. The dictionary value on the right beats the dictionary key on the left. For example, the dictionary key 'Rock' is beaten by the dictionary value 'Paper'. 

def load_image(path, camera) -> Image:
	img =

	pad ='RGB', (
		((img.size[0] + 31) // 32) * 32,
		((img.size[1] + 15) // 16) * 16,

	pad.paste(img, (0, 0))
	layer = camera.add_overlay(pad.tobytes(), size=img.size)

	return layer

Next, we include a function to load images as layers. We use these images for the countdown and to show the hand signs that the computer chooses. The function scales the images to proper size to fit on the screen.

def random_sign():
	return random.choice(signs)

def compare_signs(player_sign, computer_sign):
	if (game_logic[player_sign] == computer_sign):
		print('computer wins')
	elif (game_logic[computer_sign] == player_sign):
		print('you win')

The first function above uses random.choice to return a random element from the list of signs.

The second function compares the sign detected by the Lobe model with the sign the Raspberry Pi chose randomly. It checks whether the sign the player chose loses to the sign the computer chose.

For example, if the player chooses Rock, then game_logic[player_sign] will equal Paper since we're getting the element that is at game_logic index Rock.

Main program code

model = ImageModel.load('~/model')

ImageModel is a class from the Lobe library. We load the Lobe model and create an instance of the class.

with picamera.PiCamera(resolution=(224, 224), framerate=30) as camera:

Next, we instantiate the Pi Camera. The rest of the code runs inside this indent block.

stream = io.BytesIO()


We create a stream to continuously show the camera footage, then start the preview on the camera to populate the stream, and wait 2 seconds to let the camera warm up.

rock = load_image('assets/rock.png', camera)
paper = load_image('assets/paper.png', camera)
scissor = load_image('assets/scissor.png', camera)
counter_one = load_image('assets/one.png', camera)
counter_two = load_image('assets/two.png', camera)
counter_three = load_image('assets/three.png', camera)

The above code loads all the images we use in the game. Each image is loaded into a separate layer.

Main loop

The section below covers the main program loop.

The above code returns the stream to the first byte.

inputs = get_inputs()
while Input.BUTTON not in inputs:
	inputs = get_inputs()

Next, we wait for the button to be pressed before starting the game.

camera.preview.alpha = 0
counter_one.layer = 3

counter_one.layer = 0
counter_two.layer = 3

counter_two.layer = 0
counter_three.layer = 3

counter_three.layer = 0
camera.preview.alpha = 255

Once the button is pressed, we turn the opacity of the camera preview to 0 so it's transparent. Then we move each number of the countdown to the front and wait 1 second between each one. Finally we turn the camera opacity back to 255 (opaque).

camera.capture(stream, format='jpeg')
img =

result = model.predict(img)

label = result.prediction

camera.annotate_text = label

The next step is to capture an image from the camera stream and open it as a Pillow image. The Lobe model uses inferencing to make a prediction on the image, and returns the predicted label. This is added to the camera preview.

computer_sign = random_sign()

if (computer_sign == 'Rock'):
	Rock.layer = 3
elif (computer_sign == 'Paper'):
	Paper.layer = 3
elif (computer_sign == 'Scissors'):
	Scissors.layer = 3


Rock.layer = 0
Paper.layer = 0
Scissors.layer = 0

The above code is how the Pi plays the game! The Pi generates a random sign, checks which sign was generated, and shows the corresponding image for 2 seconds.

winner = compare_signs(label, computer_sign)

if (winner == 'player'):
	camera.annotate_text = 'You Win!'
elif (winner == 'computer'):
	camera.annotate_text = 'You Lose...'
elif (winner == 'tie'):
	camera.annotate_text = 'Tie'

Finally, we compare the player gesture (label from the prediction) and the computer gesture, check who the winner was and add the text to the camera preview.

Now that you've built a Rock, Paper, Scissors game for yourself, try making it more accessible for other folks!

The model is only as good as the data you give it. To make this model work for more people, collect images of many different hands. If you can train with hands that are feminine and masculine, old and young, small and large, and with different skin tones, you'll end up with a better model that will work for more people.

You can also use this project as a template to do other types of gesture applications and trigger different actions with the BrainCraft.

To learn how to update your Lobe model with images taken from the Pi Camera, check out this tutorial:

Happy making!

This guide was first published on Mar 31, 2021. It was last updated on 2021-03-30 15:36:50 -0400.