We're using earpatch.py to automatically create mouth movements from our .wav files. It does a fairly good job. It's not quite perfect, but it's possible to modify the file it creates to improve synching of the words and mouth.
On my Mac, I created a Rhubarb folder in my home directory, since Python on Mac requires a typing full file path, I got a bit tired of typing seven nested folder names. You can install all your software into this folder so it can work together.
Save your 16-bit .wav audio files into this folder as well.
Songs & Music Files
I wanted to play a song for my nieces as Story03.bin, but this posed a tricky problem: the earpatch.py script can't tell the difference between music and speech, so it continues to move the mouth during instrumental sections. It is also really coded for speech, not song, and it seems to get lost during singing where a vowel is held longer than it would be when speaking.
To get around this, I created two different .wav files: one with my song, and then another where I clearly spoke the lyrics in time with the song. I created the mouth.json file using the spoken .wav file, and then created the final .bin file using the singing-and-playing .wav file.
Create the Mouth Movement File
Open a terminal window and create the mouth file using a command like:
/Users/erin/Rhubarb/rhubarb -f json -o mouth_intro.json intro.wav
where the file path points to your Rhubarb folder, and intro.wav is the audio file you're converting. The script will create a file called mouth_intro.json, which we'll use when we create the .bin file on the next page of the guide.
Modifying Mouth Movement Files
Open up mouth_intro.json using a text editor and take a look at it. It should look something like this:
{
"metadata": {
"soundFile": "/Users/erin/Rhubarb/intro.wav",
"duration": 6.09
},
"mouthCues": [
{ "start": 0.00, "end": 1.23, "value": "X" },
{ "start": 1.23, "end": 1.29, "value": "C" },
{ "start": 1.29, "end": 1.48, "value": "E" },
{ "start": 1.48, "end": 1.58, "value": "A" },
{ "start": 1.58, "end": 2.28, "value": "B" },
{ "start": 2.28, "end": 2.40, "value": "A" },
{ "start": 2.40, "end": 2.51, "value": "D" },
{ "start": 2.51, "end": 2.72, "value": "B" },
{ "start": 2.72, "end": 2.86, "value": "C" },
{ "start": 2.86, "end": 3.21, "value": "B" },
{ "start": 3.21, "end": 3.83, "value": "X" },
{ "start": 3.83, "end": 4.06, "value": "F" },
{ "start": 4.06, "end": 4.13, "value": "C" },
{ "start": 4.13, "end": 4.27, "value": "F" },
{ "start": 4.27, "end": 4.62, "value": "B" },
{ "start": 4.62, "end": 4.76, "value": "E" },
{ "start": 4.76, "end": 4.97, "value": "B" },
{ "start": 4.97, "end": 6.09, "value": "X" }
]
}
Within the mouthCues section, you can add or delete lines, adjust start or end times, or change the mouth shape, called value.
While doing so, you need to make sure that:
- json syntax is respected
- the
starttimes are in ascending order - each
endtime is no later than the next "start" time - the
valueis one of the following 9 uppercase letters: A, B, C, D, E, F, G, H, X
The letters A, F, and X correspond to a closed mouth; B, C, G, and H to a partially open mouth, and D and E to a fully open mouth. With Teddy's animatronics, we don't really need this many options -- A, F, and X all do the same thing: close the mouth. I found it easier to understand and edit this file by using just one letter from each category:
- X = closed mouth
- B = partially open
- E = fully open
I opened my .wav file in Audacity to see the time points of each section, so I could find the corresponding lines in the code and modify the movements as needed.
I spent a lot of time on my intro.wav mouth file since that's the first thing kids will see, and it will get watched a lot. I didn't worry too much about the longer story files since it can get really tricky to figure out which lines need to be edited.
It helps to play the .wav file in Audacity at the same time Teddy is speaking to find the time points that need editing.
Text editor powered by tinymce.
