Machine learning is only about as complicated as cereal and marshmallows in this guide.

This project from Google uses a laptop's built-in camera to identify various cereal and marshmallows. The computer then sorts them based on a model you train. A Circuit Playground Express communicates with the computer to decide when to sort which marshmallow/cereal via a micro servo.

Parts

Circuit Playground Express is the next step towards a perfect introduction to electronics and programming. We've taken the original Circuit Playground Classic and...
$24.95
In Stock
This here is your standard A to micro-B USB cable, for USB 1.1 or 2.0. Perfect for connecting a PC to your Metro, Feather, Raspberry Pi or other dev-board or...
$2.95
In Stock
For bread-boarding with unusual non-header-friendly surfaces, these cables will be your best friends! No longer will you have long strands of alligator clips that are grabbing little...
$7.95
In Stock
Tiny little servo can rotate approximately 180 degrees (90 in each direction), and works just like the standard kinds you're used to but smaller. You can use any servo...
$5.95
In Stock

Other materials

  • Printer
  • 8.5" x 11" paper (cardstock preferred)
  • Tape or glue stick
  • Scissors or hobby knife
  • Ruler
  • Googly Eyes
  • Cereal like Lucky Charms, or other bite-sized bits that can be classified in two parts.

Optional parts (for 3D printing)

*BZZZZZZZZZZ* Feel that? That's your little buzzing motor, and for any haptic feedback project you'll want to pick up a few of them. These vibe motors are tiny discs,...
$1.95
In Stock
Simple but effective - this sewable breakout board has a CR2032 coin cell battery holder soldered on, an on/off switch and 0.1" pitch breakout pins for easy connecting. Great for...
$3.50
In Stock

Visit and print out the page linked here from the Google project site. Click the printout to download the pdf, then print it.

Cut it up

  • Cut out the two foldings.
  • For the smaller piece, you may want to consider using a hobby knife for more precision.
Please be careful with scissors or hobby knives. Adults should help younger makers with cutting items.

Form and tape together

  • Use a ruler to fold and crease along each dotted line.
  • Tape/glue together each corresponding letter.
  • Tape/glue the servo to the larger piece.
  • Tape/glue the servo arm to the smaller piece. Add the googly eyes!
  • Then the servo arm goes back on the servo.

*For more detailed visuals on the folding and taping of these parts, check out the videos in part 1C in Google's guide.

If you'd like to 3D print your sorter skip ahead to the "3D Printing" section to find out how!

Wiring

  • Connect one jumper/alligator wire from the servo's brown output onto a GND on the CPX.
  • Connect the red to the CPX 3.3V pad.
  • Connect the yellow to the A1 pad.

Now it's time to prep the Circuit Playground Express (CPX) so that your browser can recognize it and interact with the model we'll train next.

Download the UF2

Connect your Circuit Playground Express to your computer via a known, good USB cable (not a charger "power only" cable, please).

Single or double click the reset button (try both!) to enter bootloader mode.

A new flash drive should appear named CPLAYBOOT. Download the UF2 file below. Using your operating system file explorer/finder, drag the UF2 file onto the CPX's CPLAYBOOT drive.

Load TinySorter in Chrome

Ensure your Circuit Playground Express is still connected to your computer via a USB cable.

P5 is a hosting for web code that uses javascript
  • Open up this link in a new Chrome browser window.
  • Click CONNECT HARDWARE, select the board listed and hit connect.

 

Your sorter should start to move! (Disconnect the servo cable from pin A1 to stop the sorter from moving, if desired.)

  • Place the tiny sorter over your webcam so the camera can see into the bucket.
  • To create a new image project on Teachable Machine click here.

Time to grab some cereal and train some models!

Train the cereal class

Fill your first class with pictures of cereal captured from your webcam. The sorter will be shaking while training. Hint: the shaking mechanism allows the cereal to flow down the chute and into the bucket for the camera to see.

You can un-attach the pin A1 cable from the CPX in between training models to stop the sorter from shaking.

Train the marshmallow class

  • Next, fill your second class with pictures of marshmallows also captured from your webcam.
Q: How many pictures do I need to train each class?

A: Try to get somewhere between 75 - 150 pictures per class. But experiment with it! Remember the more accurate data your model has the better it will be.

Train the "nothing" class

  • Then train a third class with nothing inside the bucket. This class lets the sorter know when there is nothing in the bucket thus it should continue to shake until there is something identifiable.

Make sure to name each class corresponding to the object it represents.

To edit class names, click the pencil icon at the top of each class. 

  • Next, hit "train model" and that's it! You've trained a model.

  • Try testing out how well your model works by placing items in the bucket. The model will try to classify the items on the bottom right of the site.
  • If your model isn't recognizing very accurately, try adding more pictures to the classes that seem weak.
  • Next click "export model" and then  "upload my model" to upload to the cloud.

Load the model in the p5 sketch

  • After uploading the model, copy the sharable link to your clipboard.
  • Back on the p5 sketch site, paste in the link and hit "load model".

Test it Out.

Label a couple of bowls or cups and place next to each other accordingly under the sorter.

Drop in some cereal and see how well you've trained your model. Does it go into the right bucket?

Troubleshooting

Problem: The sorter is not accurately identifying the cereal.

Solution 1: Make sure the tiny sorter is aligned with your webcam for good classification.

Solution 2: Try adding more pictures to the "nothing" class in your model. Add some pictures of the background that your webcam sees when the sorter tilts one way or the other. This will help the model know when to keep wigglin' away! (eg. it may see the colors in your room or on your shirt and think it's a marshmallow)

Problem: The cereal is getting stuck or is moving too slowly down the ramp.

Solution: Tilt your computer screen up or down to make the ramp steeper or shallower.

You can always feel free to refer to the original guide which has videos and more.

Going further

Make this project your own! Classify whatever you can think up and train new and exciting models.

Other ideas include:

Wanna try this out with other boards?

You're in luck because Web USB and the code from this guide is now supported by any board with SAMD chip. This means you will be able to use the code from this project with the following boards:

  • PyPortal
  • Metro Express
  • Gemma M0
  • Itsy Bitsy M0 express
  • Feather M0
  • So many more!

Check out more tools and resources on machine learning from Teachable Machine.

Good luck modeling!

If you'd like to try 3D printing the sorter instead of printing and folding it out of paper, you're in luck because we designed 3D printable versions of the chute and sorter arm.

To improve the advancement of items down the chute you can add a vibrational motor to the design, which improves the feeding consistency.

Experimenter's warning: this method is not fool-proof and requires some tweaking

Download and print the below files if you'd like to experiment with the 3D printed version of the sorter.

  • Mount the servo to the sorter via the screw (comes with most micro servos).
  • Make sure the servo wires are oriented towards the top of the sorter and come down the back through the groove.
  • Lastly, attach the basket piece via tape or glue to the servo arm.

Creating a better feeding mechanism

  • The 3D printed sorter, because of it's density and weight, needs a bit more of a nudge than just the servo to move the bits down the chute. By adding a vibrating motor disc and soldering to a coin cell powered breakout, the bits can more easily flow down the chute.
  • This method is not full-proof but it does help!
  • We found that cereal and marshmallows didn't work as well as more dense items like peanut m&ms.
  • Experiment and see what works for you!
*BZZZZZZZZZZ* Feel that? That's your little buzzing motor, and for any haptic feedback project you'll want to pick up a few of them. These vibe motors are tiny discs,...
$1.95
In Stock
Simple but effective - this sewable breakout board has a CR2032 coin cell battery holder soldered on, an on/off switch and 0.1" pitch breakout pins for easy connecting. Great for...
$3.50
In Stock
A perfect match for our sew-able coin cell holder. This non-rechargeable coin cell is CR2032 sized: 20mm diameter, 3.2mm thick. It...
$0.95
In Stock
Wanna use a different board? Any board with a SAMD chip can support Web USB and thus should be able to run this code!

This guide assumes you're using a CPX, so we provide a UF2 for you to drag and drop the firmware. However, you can re-compile the code for any SAMD21/SAMD51/nRF52840 chipset that Adafruit supports with TinyUSB. That's because we have to use WebUSB to communicate and that's not easy to do with Arduino

The instructions below will show you how to edit the code to adapt or recompile it if you wish.

Set-Up Arduino

The first thing you will need to do is to download the latest release of the Arduino IDE.

Download the code

In the embedded code element below, click on the Download: Project Zip link, and save the .zip archive file to your computer.

Then, uncompress the .zip file, it will unpack to a folder named TM_WebUSB_Sorter.

/* This sketch demonstrates WebUSB as web serial with browser with WebUSB support (e.g Chrome).
 * For use with the Teachable Machine Tiny Sorter (and others!) project
 * See https://learn.adafruit.com/using-webusb-with-arduino-and-tinyusb for
 * software installation instructions and then
 * https://learn.adafruit.com/machine-learning-with-marshmallows-and-tiny-sorter
 * for usage tutorial
 * 
 * Targetted to work with Circuit Playground Express but will work with any
 * board that has TinyUSB support - don't forget to select TinyUSB in the Tools menu!
 */

#include <Servo.h>
#include <Adafruit_NeoPixel.h>
#include "Adafruit_TinyUSB.h"

// Which pin on the CPX/board is the Servo connected to?
#define SERVO_PIN A1
Servo myservo;

// Use internal neopixel ring
#define NEOPIX_PIN 8
#define NUMPIXELS  10
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(NUMPIXELS, NEOPIX_PIN, NEO_GRB + NEO_KHZ800);

// USB WebUSB object
Adafruit_USBD_WebUSB usb_web;

// Landing Page: scheme (0: http, 1: https), url
WEBUSB_URL_DEF(landingPage, 1 /*https*/, "learn.adafruit.com/machine-learning-with-marshmallows-and-tiny-sorter");

// the setup function runs once when you press reset or power the board
void setup()
{
  usb_web.begin();
  usb_web.setLandingPage(&landingPage);
  usb_web.setLineStateCallback(line_state_callback);

  Serial.begin(115200);

  // This initializes the NeoPixel with RED
  pixels.begin();
  pixels.setBrightness(20);
  pixels.fill(0x0F0F0F); // dim white
  pixels.show();

  // wait until device mounted
  while( !USBDevice.mounted() ) delay(1);
  pixels.fill(0x0F0F00); // dim yellow
  pixels.show();
  
  Serial.println("TinyUSB WebUSB RGB example");
  usb_web.print("Sketch begins.\r\n");
  usb_web.flush();
  pinMode(LED_BUILTIN, OUTPUT);

  myservo.attach(SERVO_PIN);
  myservo.write(60);  
}


void loop()
{
  if ( usb_web.available()) {
    digitalWrite(LED_BUILTIN, HIGH);
    Serial.print("-> ");
    char val = usb_web.read();
    digitalWrite(LED_BUILTIN, LOW);
    Serial.print("Read value: "); Serial.println(val, DEC);

    if (val == 1) {    // Target bin #1
       pixels.fill(0xFFFF00);
       pixels.show();
       Serial.println("CEREAL!");

       myservo.write(0);        // push cereal to one side
       delay(2000);             // wait
       for (int pos = 0; pos <= 75; pos++) {  // return servo
          myservo.write(pos);
          delay(5);
       }
       delay(1000);          // another wait before we continue

    } else if (val == 2) {    // Target bin #2
       pixels.fill(0xFF000FF);
       pixels.show();
       Serial.println("MALLOW!");

       myservo.write(180);      // push mallows to other side
       delay(2000);             // wait
       for (int pos = 180; pos >= 75; pos--) {  // return servo
          myservo.write(pos);
          delay(5);
       }
       delay(1000);          // another wait before we continue
    }
    pixels.fill(0);
    pixels.show();

    while (usb_web.available()) {
      usb_web.read();
      delay(10);
    }
  } else {
    // no webserial data, tick tock the servo
    for (int pos = 60; pos <= 90; pos++) { // slowly goes from 60 degrees to 90 degrees
      myservo.write(pos);
      delay(3);
    }
    for (int pos = 90; pos >= 60; pos--) { // goes back to 60
      myservo.write(pos);
      delay(3);
    }
  }
}

void line_state_callback(bool connected)
{
  // connected = green, disconnected = red
  pixels.fill(connected ? 0x00ff00 : 0xff0000);
  pixels.show();
}

Follow the below guide on setting up the Arduino IDE for TinyUSB with the CPX.

  • Open the TM_WebUSB_Sorter.ino sketch in Arduino IDE
  • Make sure your board is connected via USB to your computer
  • Double-check that board and port are correctly set in the tools menu, don't forget to select TinyUSB as the USB stack, you cannot use the default Arduino USB stack!
  • Then upload the sketch by hitting the upload arrow

This guide was first published on Jan 27, 2020. It was last updated on Jan 27, 2020.