PianoGlove

published July 30, 2013, last updated October 07, 2024

posted in Sensors/ Light Arduino Compatibles/ Adafruit FLORA Breakout Boards/ Storage

Intermediate

Project guide

What you'll need

The PianoGlove uses a TCS34725 to sense colors, a NeoPixel LED to 'play' those colors back visually, a VS1053 Codec board to play them back aurally, and a Flora wearable microcontroller to handle all the conversion & communication between them.

Prerequisite guides:

To build your own PianoGlove, you'll need the following items:

Flora
VS1053 Codec Breakout (includes 3.5mm jack)
Lithium Ion Polymer Battery
Flora Color Sensor
Flora RGB Smart Neo Pixel
Perma-Proto Half-sized Breadboard PCB
Fabric glove
Wire
Needle & thread
Soldering iron & solder
Wire cutters
Computer & USB cable to program Flora

In this tutorial, we'll be using an inexpensive black glove from a costume shop, but do feel free to experiment & improvise with any other types of gloves, wearables, or whatever inspires you.

Once you've gathered all your gear, we can move on to building the electronics …

Text editor powered by tinymce.

Prepare to be wired

First we'll prep the wires which run from the Flora to the Color Sensor, NeoPixel, and Codec Board.

Ribbon cable can be handy for keeping wires grouped, but any common type of hookup wire will work just fine.

Cut 7 pieces of wire a bit longer than the distance between your pointer fingertip and wrist.

Cut an additional 4 pieces of wire to about 4.5" lengths.

Strip the end of each wire and tin it with a bit of solder.

Wire Flora to Color Sensor & NeoPixel

Using the wiring diagram to the left (click to enlarge), solder the connections between the Flora and Color Sensor.

Then do the same for connections to the NeoPixel.

Be sure to connect Flora D6 to NeoPixel's input pad which is labelled with an arrow pointing toward the NeoPixel LED.

Create the Codec-Perma-Proto

The Perma-Proto board is designed to mirror the connections of a half-size breadboard. This means you can easily test your Codec board wiring temporarily on a breadboard before committing to a soldered Perma-Proto.

The Codec board comes with two strips of male header pins. Trim each of them to a length of 16 pins and place them in the Perma-Proto board as seen to the left.

Mount the Codec board on the header pins and solder them all in place. Then go ahead and solder the pins to the bottom side of the Perma-Proto.

Clip the excess pin lengths off the bottom side in order to avoid significant wrist discomfort later on!

Wire Codec board to Flora and audio jack

Solder the Codec board's included 3.5mm audio jack to the Perma-Proto in the remaining unused area.

While following along with the wiring diagram, use small pieces of solid core wire or breadboard jumpers to make on-board connections between the Codec board to the audio jack.

If you have a V2 version of the VS1053 breakout, connect the middle headphone jack pin to AGND instead of GBUF (all else is the same)

Solder connections between the Flora and Codec board using the shorter set of 4 wires you cut and tinned earlier.

Text editor powered by tinymce.

It's not necessary to wear the glove while the electronics are sewn in place, but doing so can help ensure a comfortable fit. Keep in mind, you'll need a friend with basic sewing skills and a steady hand to help out if choose to go this route.

Attach Flora

Place the Flora on the back of the glove just past the knuckle-line. Sew it in place using the unused solder pads as anchor points.

Stitch finger wiring

Wrap the NeoPixel wiring around the glove index finger so that the NeoPixel can be attached on top of the fingernail area. Secure the wires to the glove using a whip-stitch running their entire length.

Sew the NeoPixel to the back of the fingertip/fingernail area.

Wrap the Color Sensor wiring around the index finger, ensuring the sensor rests below the the face of the fingertip. Stitch the length of the wiring to the glove and sew the sensor in place.

Sew the Codec Proto

Place the Proto board at the glove's wrist area just behind the joint so that your hand can move freely. Sew it place using the any of holes on either side of the board.

Connect the battery

Turn the Flora's power switch to the off position and plug the battery into the connector opposite the USB port.

To secure the battery in place during use, it can be tucked between the Flora and glove.

Optionally, you can further secure the battery by sewing its wiring to back of the glove.

Text editor powered by tinymce.

Download the code

There are three code libraries which you'll need for this project. For modern Arduinos, please use the Manage Libraries feature to select the following libraries. For more on working with Arduino libraries, see this guide.

Once you have those two libraries loaded, you can download the code. From the program listing below, select Download File to get at the files. This will save a Zip file of the files to your computer. Remember the location you saved them to. After the download, open the Zip file and get the following files Flora_Pianoglove.h and Flora_Pianoglove.ino. You can move the Flora_Pianoglove directory to your Arduino Sketch directory and be ready to go. Just be sure both the ino and h files are in the same directory.

Download File

Copy Code

// SPDX-FileCopyrightText: 2019 Anne Barela for Adafruit Industries
//
// SPDX-License-Identifier: MIT

#include <Wire.h>
#include "Adafruit_TCS34725.h"
#include <Adafruit_NeoPixel.h>
#include "Flora_Pianoglove.h"


// we only play a note when the clear response is higher than a certain number 
#define CLEARTHRESHHOLD 2000

#define LOWTONE 1000
#define HIGHTONE 2000

// high C
#define LOWKEY 64 
// double high C
#define HIGHKEY 76

// our RGB -> eye-recognized gamma color
byte gammatable[256];

// color sensor
Adafruit_TCS34725 tcs = Adafruit_TCS34725(TCS34725_INTEGRATIONTIME_50MS, TCS34725_GAIN_4X);
// one pixel on pin 6
Adafruit_NeoPixel strip = Adafruit_NeoPixel(1, 6, NEO_GRB + NEO_KHZ800);


void setup() {
  Serial.begin(9600);
  Serial.println("Color Piano!");

  if (tcs.begin()) {
    Serial.println("Found sensor");
  } else {
    Serial.println("No TCS34725 found ... check your connections");
    while (1); // halt!
  }
  
  strip.begin();
  strip.show(); // Initialize all pixels to 'off'

  // thanks PhilB for this gamma table!
  // it helps convert RGB colors to what humans see
  for (int i=0; i<256; i++) {
    float x = i;
    x /= 255;
    x = pow(x, 2.5);
    x *= 255;
      
    gammatable[i] = x;      

    //Serial.println(gammatable[i]);
  }
  
  // tone output on OC1A / D9
  pinMode(9, OUTPUT);
  // toggle output
  TCCR1A = _BV(COM1A1);
  // PWM phase and freq correct
  TCCR1B = _BV(WGM13) | _BV(CS11);  // div by 8
  setFreq(880);
}

void setFreq(uint16_t f) {
  if (f == 0) {
    TCCR1B = 0;  // turn off
  } else {
    TCCR1B = _BV(WGM13) | _BV(CS11);   // div by 8
  }
  uint32_t i = F_CPU / 16;  // div by 8 and 2
  i /= f;
  ICR1 = i;
  OCR1A = i/2;
}

void loop() {
  uint16_t clear, red, green, blue;

  tcs.setInterrupt(false);      // turn on LED

  delay(60);  // takes 50ms to read 
  
  tcs.getRawData(&red, &green, &blue, &clear);

  tcs.setInterrupt(true);  // turn off LED

  // not close enough to colorful item
  if (clear < CLEARTHRESHHOLD) {
    setFreq(0);
    strip.setPixelColor(0, strip.Color(0, 0, 0)); // turn off the LED
    strip.show();
    return;
  }
  
  Serial.print("C:\t"); Serial.print(clear);
  Serial.print("\tR:\t"); Serial.print(red);
  Serial.print("\tG:\t"); Serial.print(green);
  Serial.print("\tB:\t"); Serial.print(blue);

  // Figure out some basic hex code for visualization
  uint32_t sum = red;
  sum += green;
  sum += blue;
  sum = clear;
  float r, g, b;
  r = red; r /= sum;
  g = green; g /= sum;
  b = blue; b /= sum;
  r *= 256; g *= 256; b *= 256;
  if (r > 255) r = 255;
  if (g > 255) g = 255;
  if (b > 255) b = 255;
  
  Serial.print("\t");
  Serial.print((int)r, HEX); Serial.print((int)g, HEX); Serial.print((int)b, HEX); 
  Serial.println();
 
  
  // OK we have to find the two primary colors
  // check if blue is smallest. MEME: fix for 'white'
  float remove, normalize;
  if ((b < g) && (b < r)) {
    remove = b;
    normalize = max(r-b, g-b);
  } else if ((g < b) && (g < r)) {
    remove = g;
    normalize = max(r-g, b-g);
  } else {
    remove = r;
    normalize = max(b-r, g-r);
  }
  // get rid of minority report
  float rednorm = r - remove;
  float greennorm = g - remove;
  float bluenorm = b - remove;
  // now normalize for the highest number
  rednorm /= normalize;
  greennorm /= normalize;
  bluenorm /= normalize;

  Serial.println();
  strip.setPixelColor(0, strip.Color(gammatable[(int)r], gammatable[(int)g], gammatable[(int)b]));
  strip.show();

  Serial.print(rednorm); Serial.print(", "); 
  Serial.print(greennorm); Serial.print(", "); 
  Serial.print(bluenorm); Serial.print(" "); 
  Serial.println();

  float rainbowtone = 0;
  
  if (bluenorm <= 0.1) {
    // between red and green
    if (rednorm >= 0.99) {
      // between red and yellow
      rainbowtone = 0 + 0.2 * greennorm;
    } else {
      // between yellow and green
      rainbowtone = 0.2 + 0.2 * (1.0 - rednorm);
    }
  } else if (rednorm <= 0.1) {
    // between green and blue
    if (greennorm >= 0.99) {
      // between green and teal
      rainbowtone = 0.4 + 0.2 * bluenorm;
    } else {
      // between teal and blue
      rainbowtone = 0.6 + 0.2 * (1.0 - greennorm);
    }
  } else {
    // between blue and violet
    if (bluenorm >= 0.99) {
      // between blue and violet
      rainbowtone = 0.8 + 0.2 * rednorm;
    } else {
      // between teal and blue
      rainbowtone = 0; 
    }
  }
  
  Serial.print("Scalar "); Serial.println(rainbowtone);
  float keynum = LOWKEY + (HIGHKEY - LOWKEY) * rainbowtone;
  Serial.print("Key #"); Serial.println(keynum);
  float freq = pow(2, (keynum - 49) / 12.0) * 440;
  Serial.print("Freq = "); Serial.println(freq);  
  //Serial.print((int)r ); Serial.print(" "); Serial.print((int)g);Serial.print(" ");  Serial.println((int)b );
  setFreq(freq);
}



RgbColor HsvToRgb(HsvColor hsv)
{
    RgbColor rgb;
    unsigned char region, remainder, p, q, t;

    if (hsv.s == 0)
    {
        rgb.r = hsv.v;
        rgb.g = hsv.v;
        rgb.b = hsv.v;
        return rgb;
    }

    region = hsv.h / 43;
    remainder = (hsv.h - (region * 43)) * 6; 

    p = (hsv.v * (255 - hsv.s)) >> 8;
    q = (hsv.v * (255 - ((hsv.s * remainder) >> 8))) >> 8;
    t = (hsv.v * (255 - ((hsv.s * (255 - remainder)) >> 8))) >> 8;

    switch (region)
    {
        case 0:
            rgb.r = hsv.v; rgb.g = t; rgb.b = p;
            break;
        case 1:
            rgb.r = q; rgb.g = hsv.v; rgb.b = p;
            break;
        case 2:
            rgb.r = p; rgb.g = hsv.v; rgb.b = t;
            break;
        case 3:
            rgb.r = p; rgb.g = q; rgb.b = hsv.v;
            break;
        case 4:
            rgb.r = t; rgb.g = p; rgb.b = hsv.v;
            break;
        default:
            rgb.r = hsv.v; rgb.g = p; rgb.b = q;
            break;
    }

    return rgb;
}

HsvColor RgbToHsv(RgbColor rgb)
{
    HsvColor hsv;
    unsigned char rgbMin, rgbMax;

    rgbMin = rgb.r < rgb.g ? (rgb.r < rgb.b ? rgb.r : rgb.b) : (rgb.g < rgb.b ? rgb.g : rgb.b);
    rgbMax = rgb.r > rgb.g ? (rgb.r > rgb.b ? rgb.r : rgb.b) : (rgb.g > rgb.b ? rgb.g : rgb.b);

    hsv.v = rgbMax;
    if (hsv.v == 0)
    {
        hsv.h = 0;
        hsv.s = 0;
        return hsv;
    }

    hsv.s = 255 * long(rgbMax - rgbMin) / hsv.v;
    if (hsv.s == 0)
    {
        hsv.h = 0;
        return hsv;
    }

    if (rgbMax == rgb.r)
        hsv.h = 0 + 43 * (rgb.g - rgb.b) / (rgbMax - rgbMin);
    else if (rgbMax == rgb.g)
        hsv.h = 85 + 43 * (rgb.b - rgb.r) / (rgbMax - rgbMin);
    else
        hsv.h = 171 + 43 * (rgb.r - rgb.g) / (rgbMax - rgbMin);

    return hsv;
}

// SPDX-FileCopyrightText: 2019 Anne Barela for Adafruit Industries
//
// SPDX-License-Identifier: MIT

#include <Wire.h>
#include "Adafruit_TCS34725.h"
#include <Adafruit_NeoPixel.h>
#include "Flora_Pianoglove.h"


// we only play a note when the clear response is higher than a certain number 
#define CLEARTHRESHHOLD 2000

#define LOWTONE 1000
#define HIGHTONE 2000

// high C
#define LOWKEY 64 
// double high C
#define HIGHKEY 76

// our RGB -> eye-recognized gamma color
byte gammatable[256];

// color sensor
Adafruit_TCS34725 tcs = Adafruit_TCS34725(TCS34725_INTEGRATIONTIME_50MS, TCS34725_GAIN_4X);
// one pixel on pin 6
Adafruit_NeoPixel strip = Adafruit_NeoPixel(1, 6, NEO_GRB + NEO_KHZ800);


void setup() {
  Serial.begin(9600);
  Serial.println("Color Piano!");

  if (tcs.begin()) {
    Serial.println("Found sensor");
  } else {
    Serial.println("No TCS34725 found ... check your connections");
    while (1); // halt!
  }
  
  strip.begin();
  strip.show(); // Initialize all pixels to 'off'

  // thanks PhilB for this gamma table!
  // it helps convert RGB colors to what humans see
  for (int i=0; i<256; i++) {
    float x = i;
    x /= 255;
    x = pow(x, 2.5);
    x *= 255;
      
    gammatable[i] = x;      

    //Serial.println(gammatable[i]);
  }
  
  // tone output on OC1A / D9
  pinMode(9, OUTPUT);
  // toggle output
  TCCR1A = _BV(COM1A1);
  // PWM phase and freq correct
  TCCR1B = _BV(WGM13) | _BV(CS11);  // div by 8
  setFreq(880);
}

void setFreq(uint16_t f) {
  if (f == 0) {
    TCCR1B = 0;  // turn off
  } else {
    TCCR1B = _BV(WGM13) | _BV(CS11);   // div by 8
  }
  uint32_t i = F_CPU / 16;  // div by 8 and 2
  i /= f;
  ICR1 = i;
  OCR1A = i/2;
}

void loop() {
  uint16_t clear, red, green, blue;

  tcs.setInterrupt(false);      // turn on LED

  delay(60);  // takes 50ms to read 
  
  tcs.getRawData(&red, &green, &blue, &clear);

  tcs.setInterrupt(true);  // turn off LED

  // not close enough to colorful item
  if (clear < CLEARTHRESHHOLD) {
    setFreq(0);
    strip.setPixelColor(0, strip.Color(0, 0, 0)); // turn off the LED
    strip.show();
    return;
  }
  
  Serial.print("C:\t"); Serial.print(clear);
  Serial.print("\tR:\t"); Serial.print(red);
  Serial.print("\tG:\t"); Serial.print(green);
  Serial.print("\tB:\t"); Serial.print(blue);

  // Figure out some basic hex code for visualization
  uint32_t sum = red;
  sum += green;
  sum += blue;
  sum = clear;
  float r, g, b;
  r = red; r /= sum;
  g = green; g /= sum;
  b = blue; b /= sum;
  r *= 256; g *= 256; b *= 256;
  if (r > 255) r = 255;
  if (g > 255) g = 255;
  if (b > 255) b = 255;
  
  Serial.print("\t");
  Serial.print((int)r, HEX); Serial.print((int)g, HEX); Serial.print((int)b, HEX); 
  Serial.println();
 
  
  // OK we have to find the two primary colors
  // check if blue is smallest. MEME: fix for 'white'
  float remove, normalize;
  if ((b < g) && (b < r)) {
    remove = b;
    normalize = max(r-b, g-b);
  } else if ((g < b) && (g < r)) {
    remove = g;
    normalize = max(r-g, b-g);
  } else {
    remove = r;
    normalize = max(b-r, g-r);
  }
  // get rid of minority report
  float rednorm = r - remove;
  float greennorm = g - remove;
  float bluenorm = b - remove;
  // now normalize for the highest number
  rednorm /= normalize;
  greennorm /= normalize;
  bluenorm /= normalize;

  Serial.println();
  strip.setPixelColor(0, strip.Color(gammatable[(int)r], gammatable[(int)g], gammatable[(int)b]));
  strip.show();

  Serial.print(rednorm); Serial.print(", "); 
  Serial.print(greennorm); Serial.print(", "); 
  Serial.print(bluenorm); Serial.print(" "); 
  Serial.println();

  float rainbowtone = 0;
  
  if (bluenorm <= 0.1) {
    // between red and green
    if (rednorm >= 0.99) {
      // between red and yellow
      rainbowtone = 0 + 0.2 * greennorm;
    } else {
      // between yellow and green
      rainbowtone = 0.2 + 0.2 * (1.0 - rednorm);
    }
  } else if (rednorm <= 0.1) {
    // between green and blue
    if (greennorm >= 0.99) {
      // between green and teal
      rainbowtone = 0.4 + 0.2 * bluenorm;
    } else {
      // between teal and blue
      rainbowtone = 0.6 + 0.2 * (1.0 - greennorm);
    }
  } else {
    // between blue and violet
    if (bluenorm >= 0.99) {
      // between blue and violet
      rainbowtone = 0.8 + 0.2 * rednorm;
    } else {
      // between teal and blue
      rainbowtone = 0; 
    }
  }
  
  Serial.print("Scalar "); Serial.println(rainbowtone);
  float keynum = LOWKEY + (HIGHKEY - LOWKEY) * rainbowtone;
  Serial.print("Key #"); Serial.println(keynum);
  float freq = pow(2, (keynum - 49) / 12.0) * 440;
  Serial.print("Freq = "); Serial.println(freq);  
  //Serial.print((int)r ); Serial.print(" "); Serial.print((int)g);Serial.print(" ");  Serial.println((int)b );
  setFreq(freq);
}



RgbColor HsvToRgb(HsvColor hsv)
{
    RgbColor rgb;
    unsigned char region, remainder, p, q, t;

    if (hsv.s == 0)
    {
        rgb.r = hsv.v;
        rgb.g = hsv.v;
        rgb.b = hsv.v;
        return rgb;
    }

    region = hsv.h / 43;
    remainder = (hsv.h - (region * 43)) * 6; 

    p = (hsv.v * (255 - hsv.s)) >> 8;
    q = (hsv.v * (255 - ((hsv.s * remainder) >> 8))) >> 8;
    t = (hsv.v * (255 - ((hsv.s * (255 - remainder)) >> 8))) >> 8;

    switch (region)
    {
        case 0:
            rgb.r = hsv.v; rgb.g = t; rgb.b = p;
            break;
        case 1:
            rgb.r = q; rgb.g = hsv.v; rgb.b = p;
            break;
        case 2:
            rgb.r = p; rgb.g = hsv.v; rgb.b = t;
            break;
        case 3:
            rgb.r = p; rgb.g = q; rgb.b = hsv.v;
            break;
        case 4:
            rgb.r = t; rgb.g = p; rgb.b = hsv.v;
            break;
        default:
            rgb.r = hsv.v; rgb.g = p; rgb.b = q;
            break;
    }

    return rgb;
}

HsvColor RgbToHsv(RgbColor rgb)
{
    HsvColor hsv;
    unsigned char rgbMin, rgbMax;

    rgbMin = rgb.r < rgb.g ? (rgb.r < rgb.b ? rgb.r : rgb.b) : (rgb.g < rgb.b ? rgb.g : rgb.b);
    rgbMax = rgb.r > rgb.g ? (rgb.r > rgb.b ? rgb.r : rgb.b) : (rgb.g > rgb.b ? rgb.g : rgb.b);

    hsv.v = rgbMax;
    if (hsv.v == 0)
    {
        hsv.h = 0;
        hsv.s = 0;
        return hsv;
    }

    hsv.s = 255 * long(rgbMax - rgbMin) / hsv.v;
    if (hsv.s == 0)
    {
        hsv.h = 0;
        return hsv;
    }

    if (rgbMax == rgb.r)
        hsv.h = 0 + 43 * (rgb.g - rgb.b) / (rgbMax - rgbMin);
    else if (rgbMax == rgb.g)
        hsv.h = 85 + 43 * (rgb.b - rgb.r) / (rgbMax - rgbMin);
    else
        hsv.h = 171 + 43 * (rgb.r - rgb.g) / (rgbMax - rgbMin);

    return hsv;
}

Download File

Copy Code

// SPDX-FileCopyrightText: 2019 Anne Barela for Adafruit Industries
//
// SPDX-License-Identifier: MIT

typedef struct RgbColor
{
    unsigned char r;
    unsigned char g;
    unsigned char b;
} RgbColor;

typedef struct HsvColor
{
    unsigned char h;
    unsigned char s;
    unsigned char v;
} HsvColor;

// SPDX-FileCopyrightText: 2019 Anne Barela for Adafruit Industries
//
// SPDX-License-Identifier: MIT

typedef struct RgbColor
{
    unsigned char r;
    unsigned char g;
    unsigned char b;
} RgbColor;

typedef struct HsvColor
{
    unsigned char h;
    unsigned char s;
    unsigned char v;
} HsvColor;

Upload the sketch

For more information on programming your Flora board including the software you need to do so, head over to the Getting Started with Flora guide.

Download & open the PianoGlove MIDI sketch in the Adafruit-Arduino IDE

Connect Flora via USB to your computer

Set Adafruit-Arduino IDE's board setting to "Flora" & upload the sketch.

Play some colors!

Once the sketch has been uploaded to the Flora, wait for the onboard red LED to stop flashing, then disconnect the USB cable.

Use Flora's power switch to turn on battery power. After a short load time, the Flora's red LED should go dark and Color Sensor's white LED should now be on.

Connect a powered speaker or headphones to the 3.5mm audio jack and move your new color-sensing index finger close to something colorful to play a musical note through the audio jack.

Try moving the sensor quickly between different bright/dark colors, or 'gate' a note by quickly pulling the sensor away from an object. Experiment with … well, anything that has color!