# How to train new TensorFlow Lite micro speech models

## Overview

![](https://cdn-learn.adafruit.com/assets/assets/000/078/277/medium800/biometric_mltf.png?1563337608)

Machine learning has come to the 'edge' - small microcontrollers that can run a very miniature version of TensorFlow Lite to do ML computations. The first demos available are for 'micro speech' which is detecting a couple words. The default words are 'yes/no' but the dataset contains many other words! This guide goes through how to train micro speech models on your own

# How to train new TensorFlow Lite micro speech models

## Training with Colab

Primary: 

# Getting Started

**[To get started, click here to open our notebook in Colab.](https://colab.research.google.com/drive/1cR5JJNYzWLlgSK9c16nnFLhxtW_bmO-p#sandboxMode=true&readOnly=true)**

When you click the link, it should take you to a page that looks like this:

![](https://cdn-learn.adafruit.com/assets/assets/000/083/818/medium800/biometric_guide-large.png?1573153094 The notebook looks like this when you first open it.)

# Configuring your Colab instance

To get started, move your mouse cursor over the `[  ]` box to the left of the first code snippet, underneath the **Configure training** header. It will change to a "Play" icon. Click on this icon.

![](https://cdn-learn.adafruit.com/assets/assets/000/083/822/medium800/biometric_run-cell.png?1573153276)

Info: 

# What should happen?

After typically 20 seconds or so, you'll see the notebook come to life. The previous output will vanish and you'll see it replaced with the result of running on your new *runtime* (see the section titled*Aside* below for more about what a *runtime* is).

When you see the following output, you know you've finished this step. You can open another copy of the notebook and compare it to our previous run, just to make sure it looks correct.	

```
Training these words: ['yes', 'no']
Training steps in each stage: [15000, 3000]
Learning rate in each stage: ['0.001', '0.0001']
Total number of training steps: 18000
```

## Aside: Behind the Scenes
Each time you open a Colab notebook, Google lets you temporarily use a computer in their datacenter to run your code. This computer is running a program called the **runtime**, which lets you play around with TensorFlow without having to worry about how fast your computer and without needing to buy an expensive graphics card.

When you close your Colab notebook, Google replaces your *runtime* with a brand new one, and releases your machine to someone else. This means that each time you come back, you'll need to set up the machine from scratch.

The first few cells in the notebook do just that.

# Connect to Google Drive

As mentioned in the last **Aside** section, the **runtime** (along with any files created) is lost when you close your browser tab. We'll need to find somewhere more permanent to store our trained model. Fortunately, Google provides a handy way to connect your Google Drive to the notebook. It appears just like a regular folder on the Colab runtime.

The next cell will connect your Google Drive to the Colab. You'll need to authorize the connection in a new browser tab. First, run the next cell, titled **Connect to Google Drive**. You should see something like this appear:

![](https://cdn-learn.adafruit.com/assets/assets/000/083/827/medium800/biometric_drive.png?1573154602 You'll need to authorize Colab to use your Google Drive. We use this to store files more permanently, since Colab only provides temporary storage.)

Click the URL, which will open a new browser tab. Go through the steps to select the Google account you want to use, and allow access to the **Google Drive File Stream** app. This was written by Google to work with Colab. 

When you've successfully authorized it, you'll see this screen (the code in this image is blurred). Copy the code (or click the helpful copy button to the right), switch back to the Colab tab, and paste the code into the text box below the *Enter your authorization code* text.

![](https://cdn-learn.adafruit.com/assets/assets/000/083/828/medium800/biometric_drive-auth.png?1573155356 When you reach this screen, you've authorized Colab to use your Google Drive. Now, copy and paste the code back into the notebook.)

# Training the model

Now you're ready to train your speech recogntion model! Run the next few cells, titled **Install Dependencies** and **Download Tensorflow**.

If you want to visualize training while it's in progress, run the **Optional: Visualize graph and training rate** cell. This isn't required, though.

Finally, run the **Create trained model** cell. This will run for several hours, and you can't close your browser tab--so, be sure you can leave your computer running for a while.

**Model Output**

You can find your model output on your [Google Drive](https://drive.google.com), in a folder called `speech-recognition`. You should see something like the following screen.

![](https://cdn-learn.adafruit.com/assets/assets/000/083/829/medium800/biometric_drive-output.png?1573155771)

# How to train new TensorFlow Lite micro speech models

## Install Docker

Warning: 

We need to be able to run a specific version/commit of TensorFlow and the dependancy requirements for TF are very extreme. We strongly suggest _against_ trying to compile and run on your native computer OS - that way we don't get weird interactions with your OS, compiler toolchain, Python kit, etc. Also, TF really wants to run on a particular version of Linux and chances are you aren't running it.

Instead, we will be using Docker to containerize and separate the TF build so we have a compact, clean, dependable build. Docker is lighter than VMWare/vagrant, and has a very nice 'hub' backend for saving/restoring your images, all for free!

# Signup and log into Docker

Sign up at [https://hub.docker.com/signup](https://hub.docker.com/signup)

 

You don't need to pay for an account, but be aware the software images we'll be using are public so don't put any private data in em!

![biometric_image.png](https://cdn-learn.adafruit.com/assets/assets/000/077/651/medium640/biometric_image.png?1561938902)

# Download and Install Desktop Docker

Download Docker software for Windows or Mac, whichever matches your computer

![biometric_image.png](https://cdn-learn.adafruit.com/assets/assets/000/077/652/medium640/biometric_image.png?1561939052)

# TensorFlow needs **a lot** of computing resources

**Give it as many CPUs and as much RAM as you can spare**

You need to give it _at least **8 GB** _ of RAM or gcc will fail with a very annoying and somewhat confusing error like this (but on some other file)

![biometric_image.png](https://cdn-learn.adafruit.com/assets/assets/000/077/654/medium640/biometric_image.png?1561939541)

```auto
ERROR: /root/tensorflow/tensorflow/core/kernels/BUILD:3371:1: C++ compilation of rule '//tensorflow/core/kernels:reduction_ops' failed (Exit 4)
gcc: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
Target //tensorflow/examples/speech_commands:train failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6058.951s, Critical Path: 3278.24s
INFO: 2606 processes: 2606 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
```

Open a command terminal and try to login, use the same username/password as from the site

![](https://cdn-learn.adafruit.com/assets/assets/000/077/655/medium800/biometric_image.png?1561940143)

OK you're ready to go!

# How to train new TensorFlow Lite micro speech models

## Create/Fork Docker Image

Start with the official TensorFlow Docker image, like github you can `pull`/`commit`/`push` and implictly fork when you do this between sources.

`docker pull tensorflow/tensorflow` will get you the latest docker image from Google

![](https://cdn-learn.adafruit.com/assets/assets/000/078/189/medium800/biometric_1_dockpull.png?1563137127)

![](https://cdn-learn.adafruit.com/assets/assets/000/078/190/medium800/biometric_2_pulled.png?1563137141)

Log into the Docker image with

`docker run -it tensorflow/tensorflow bash`

![](https://cdn-learn.adafruit.com/assets/assets/000/078/191/medium800/biometric_3_runit.png?1563137164)

Within the Docker root shell, install some dependencies with

`apt-get install -y curl zip git`

![](https://cdn-learn.adafruit.com/assets/assets/000/078/192/medium800/biometric_4_aptget.png?1563137184)

# How to train new TensorFlow Lite micro speech models

## Advanced: Build TensorFlow

If you need to compile TensorFlow from scratch, you can do it, but its very slow to get everything compiled. Once its compiled, its really fast to train models!

We have to start this way,  until there's more automated methods...so here's a guide on how we did it

Danger: 

 **We need to use version 0.23.1 of bazel** (the build tool), so we'll install that specific version like this:

`cd ~`  
`curl -O -L https://github.com/bazelbuild/bazel/releases/download/0.23.1/bazel-0.23.1-installer-linux-x86_64.sh`  
`chmod +x bazel-0.23.1-installer-linux-x86_64.sh`  
`./bazel-0.23.1-installer-linux-x86_64.sh`

![](https://cdn-learn.adafruit.com/assets/assets/000/077/660/medium800/biometric_5_installbazel.png?1561940466)

![](https://cdn-learn.adafruit.com/assets/assets/000/077/661/medium800/biometric_6_bazelinstalled.png?1561940513)

You can verify it with `bazel version`

![](https://cdn-learn.adafruit.com/assets/assets/000/077/666/medium800/biometric_image.png?1561940999)

For some reason, the image is still using Python 2.7, so grab the future package so we can run python3 code

`pip install future`

We also need to get the right version of the 'estimator' package (we use it later)

`pip uninstall tensorflow_estimator`

`pip install -I tensorflow_estimator==1.13.0 `

![](https://cdn-learn.adafruit.com/assets/assets/000/077/662/medium800/biometric_7_pipfuture.png?1561940595)

![](https://cdn-learn.adafruit.com/assets/assets/000/077/667/medium800/biometric_image.png?1561941157)

[We need to build a specific commit of TensorFlow](https://github.com/tensorflow/tensorflow/commit/4a464440b2e8f382f442b6e952d64a56701ab045), so clone the repo then switch to that commit

`git clone https://github.com/tensorflow/tensorflow.git`  
`cd tensorflow`  
`git checkout 4a464440b2e8f382f442b6e952d64a56701ab045`

![](https://cdn-learn.adafruit.com/assets/assets/000/077/663/medium800/biometric_8_gitclone.png?1561940699)

![](https://cdn-learn.adafruit.com/assets/assets/000/077/664/medium800/biometric_9_gitcheck.png?1561940703)

Go with the default configuration by running

`yes "" | ./configure`

![](https://cdn-learn.adafruit.com/assets/assets/000/077/665/medium800/biometric_10_bazelconfig.png?1561940765)

Finally start the TensorFlow compile and speech training with

`bazel run -c opt --copt=-mavx2 --copt=-mfma tensorflow/examples/speech_commands:train -- --model_architecture=tiny_conv --window_stride=20 --preprocess=micro --wanted_words="yes,no" --silence_percentage=25 --unknown_percentage=25 --quantize=1`

This will create a micro model of the large speech data set with only "yes" and "no" words in the model (to keep it small/simple)

![](https://cdn-learn.adafruit.com/assets/assets/000/077/668/medium800/biometric_bazelrun.png?1561941574)

 **This will take many hours especially the first time!** Go take a break and do something else (or, you can try using your computer but it will be slow because Docker is sucking up all the computational resources to compile 16,000 files)

![](https://cdn-learn.adafruit.com/assets/assets/000/077/669/medium800/biometric_image.png?1561941654)

![](https://cdn-learn.adafruit.com/assets/assets/000/077/671/medium800/biometric_image.png?1561949311)

After TensorFlow has completed compiling it will take another 2+ hours to run the training. In the end you will get something like this:

![](https://cdn-learn.adafruit.com/assets/assets/000/077/672/medium800/biometric_image.png?1561957237)

# How to train new TensorFlow Lite micro speech models

## Training and freezing models

Start training a new micro speech model with

`python tensorflow/examples/speech_commands/train.py -- --model_architecture=tiny_conv --window_stride=20 --preprocess=micro --wanted_words="yes,no" --silence_percentage=25 --unknown_percentage=25 --quantize=1`

or, if using bazel

`bazel run -c opt --copt=-mavx2 --copt=-mfma tensorflow/examples/speech_commands:train -- --model_architecture=tiny_conv --window_stride=20 --preprocess=micro --wanted_words="yes,no" --silence_percentage=25 --unknown_percentage=25 --quantize=1`

![](https://cdn-learn.adafruit.com/assets/assets/000/078/197/medium800/biometric_image.png?1563137932)

This will run for a few hours

![](https://cdn-learn.adafruit.com/assets/assets/000/078/199/medium800/biometric_image.png?1563142247)

At the end you'll get your final test accuracy and checkpoint file

![](https://cdn-learn.adafruit.com/assets/assets/000/078/200/medium800/biometric_image.png?1563204813)

Checkpoint files are stored in /tmp

![](https://cdn-learn.adafruit.com/assets/assets/000/078/201/medium800/biometric_image.png?1563204929)

In this case we want **/tmp/speech\_commands\_train/conv.ckpt-18000.\*** (the last place the trainer saved to)

![](https://cdn-learn.adafruit.com/assets/assets/000/078/202/medium800/biometric_image.png?1563204973)

# Freeze

Take the trained weights and turn them into a frozen model on disk.

```
python tensorflow/examples/speech_commands/freeze.py --model_architecture=tiny_conv --window_stride=20 --preprocess=micro --wanted_words="yes,no" --quantize=1 --output_file=/tmp/tiny_conv.pb --start_checkpoint=/tmp/speech_commands_train/conv.ckpt-100
```

or if using bazel something like:

```
bazel run tensorflow/examples/speech_commands:freeze -- --model_architecture=tiny_conv --window_stride=20 --preprocess=micro --wanted_words="yes,no" --quantize=1 --output_file=/tmp/tiny_conv.pb --start_checkpoint=/tmp/speech_commands_train/tiny_conv.ckpt-18000
```
![](https://cdn-learn.adafruit.com/assets/assets/000/078/268/medium800/biometric_image.png?1563324693)

![](https://cdn-learn.adafruit.com/assets/assets/000/078/269/medium800/biometric_image.png?1563324708)

# Convert

Convert the TensorFlow model into a TF Lite file

```
bazel run tensorflow/lite/toco:toco -- --input_file=/tmp/tiny_conv.pb --output_file=/tmp/tiny_conv.tflite --input_shapes=1,49,40,1 --input_arrays=Reshape_1 --output_arrays='labels_softmax' --inference_type=QUANTIZED_UINT8 --mean_values=0 --std_values=9.8077
```
![](https://cdn-learn.adafruit.com/assets/assets/000/078/270/medium800/biometric_image.png?1563325193)

The file can now be found in **/tmp/tiny\_conf.tflite**

![](https://cdn-learn.adafruit.com/assets/assets/000/078/271/medium800/biometric_image.png?1563325244)

# Extract & Save

Finally, you can use **docker cp** to copy the file from your container to your desktop. From the host computer (_not the docker contrainer_) run **docker cp CONTAINERID:/tmp/tiny\_conf.tflite .**

You should now have access to the file!

![](https://cdn-learn.adafruit.com/assets/assets/000/078/273/medium800/biometric_image.png?1563325554)

Here are some example files

[updown.zip](https://cdn-learn.adafruit.com/assets/assets/000/078/274/original/updown.zip?1563335072)
[yesno.zip](https://cdn-learn.adafruit.com/assets/assets/000/078/275/original/yesno.zip?1563335096)
[catdog.zip](https://cdn-learn.adafruit.com/assets/assets/000/078/276/original/catdog.zip?1563335165)
# How to train new TensorFlow Lite micro speech models

## Commit Docker

Now's a good time to 'save' our work. Run `docker ps` to list all your docker containers

![](https://cdn-learn.adafruit.com/assets/assets/000/077/673/medium800/biometric_image.png?1561957292)

you can 'save' this docker container to your account with

` docker commit CONTAINER_ID USERNAME/mytensorflow`

where the CONTAINER\_ID is the 12 character that is to the left of the image name and USERNAME is your docker login name. So in my case, `docker commit c2a0a7f0a7bb ladyada/mytensorflow`

It will take a few minutes while Docker runs, eventually you'll get this on the terminal:

![](https://cdn-learn.adafruit.com/assets/assets/000/077/675/medium800/biometric_image.png?1561961409)

Then push it to docker hub with **docker push username/containername**

![](https://cdn-learn.adafruit.com/assets/assets/000/077/676/medium800/biometric_image.png?1561961437)

Then visit your dockerhub profile to see that you have in fact pushed the docker image

![](https://cdn-learn.adafruit.com/assets/assets/000/078/272/medium800/biometric_image.png?1563325402)


## Featured Products

### Adafruit Metro M4 feat. Microchip ATSAMD51

[Adafruit Metro M4 feat. Microchip ATSAMD51](https://www.adafruit.com/product/3382)
Are you ready? Really ready? Cause here comes the fastest, most powerful Metro ever. The **Adafruit Metro M4** featuring the **Microchip ATSAMD51**. This Metro is like a bullet train, with its 120MHz Cortex M4 with floating point support. Your code will zig and zag...

In Stock
[Buy Now](https://www.adafruit.com/product/3382)
[Related Guides to the Product](https://learn.adafruit.com/products/3382/guides)
### Adafruit Feather M4 Express - Featuring ATSAMD51

[Adafruit Feather M4 Express - Featuring ATSAMD51](https://www.adafruit.com/product/3857)
It's what you've been waiting for, the Feather M4 Express featuring ATSAMD51. This Feather is fast like a swift, smart like an owl, strong like a ox-bird (it's half ox, half bird, OK?) This feather is powered by our new favorite chip, the **ATSAMD51J19** -  with...

In Stock
[Buy Now](https://www.adafruit.com/product/3857)
[Related Guides to the Product](https://learn.adafruit.com/products/3857/guides)
### Adafruit Metro M4 Express AirLift (WiFi) - Lite

[Adafruit Metro M4 Express AirLift (WiFi) - Lite](https://www.adafruit.com/product/4000)
Give your next project a lift with _AirLift_ - our witty name for the ESP32 co-processor that graces this Metro M4. You already know about the  **Adafruit Metro M4**  featuring the  **Microchip ATSAMD51** , with it's 120MHz Cortex M4 with...

In Stock
[Buy Now](https://www.adafruit.com/product/4000)
[Related Guides to the Product](https://learn.adafruit.com/products/4000/guides)
### Adafruit PyGamer for MakeCode Arcade, CircuitPython or Arduino

[Adafruit PyGamer for MakeCode Arcade, CircuitPython or Arduino](https://www.adafruit.com/product/4242)
What fits in your pocket, is fully Open Source, and can run CircuitPython, MakeCode Arcade or Arduino games you write yourself? That's right, it's the **Adafruit PyGamer!** We wanted to make an entry-level gaming handheld for DIY gaming, and maybe a little...

Out of Stock
[Buy Now](https://www.adafruit.com/product/4242)
[Related Guides to the Product](https://learn.adafruit.com/products/4242/guides)
### Adafruit PyBadge for MakeCode Arcade, CircuitPython, or Arduino

[Adafruit PyBadge for MakeCode Arcade, CircuitPython, or Arduino](https://www.adafruit.com/product/4200)
What's the size of a credit card and can run CircuitPython, MakeCode Arcade or Arduino? That's right, its the **Adafruit PyBadge!** We wanted to see how much we could cram into a ​3 3⁄8 × ​2 1⁄8 inch rounded rectangle, to make an all-in-one dev board with...

Out of Stock
[Buy Now](https://www.adafruit.com/product/4200)
[Related Guides to the Product](https://learn.adafruit.com/products/4200/guides)
### Adafruit PyBadge LC - MakeCode Arcade, CircuitPython, or Arduino

[Adafruit PyBadge LC - MakeCode Arcade, CircuitPython, or Arduino](https://www.adafruit.com/product/3939)
What's the size of a credit card and can run CircuitPython, MakeCode Arcade or Arduino even when you're on a budget? That's right, it's the  **Adafruit PyBadge LC!**  We wanted to see how much we could cram into a ​3 3⁄8 × ​2 1⁄8 inch...

In Stock
[Buy Now](https://www.adafruit.com/product/3939)
[Related Guides to the Product](https://learn.adafruit.com/products/3939/guides)
### Adafruit ItsyBitsy M4 Express featuring ATSAMD51

[Adafruit ItsyBitsy M4 Express featuring ATSAMD51](https://www.adafruit.com/product/3800)
What's smaller than a Feather but larger than a Trinket? It's an **Adafruit ItsyBitsy M4 Express** featuring the **Microchip ATSAMD51**! Small, powerful, with a ultra fast ATSAMD51 Cortex M4 processor running at 120 MHz - this microcontroller board is perfect...

Out of Stock
[Buy Now](https://www.adafruit.com/product/3800)
[Related Guides to the Product](https://learn.adafruit.com/products/3800/guides)

## Related Guides

- [Adafruit Metro M4 Express featuring ATSAMD51](https://learn.adafruit.com/adafruit-metro-m4-express-featuring-atsamd51.md)
- [Adafruit Feather M4 Express](https://learn.adafruit.com/adafruit-feather-m4-express-atsamd51.md)
- [Adafruit Metro M4 Express AirLift (WiFi)](https://learn.adafruit.com/adafruit-metro-m4-express-airlift-wifi.md)
- [Adafruit PyBadge and PyBadge LC](https://learn.adafruit.com/adafruit-pybadge.md)
- [Introducing Adafruit PyGamer](https://learn.adafruit.com/adafruit-pygamer.md)
- [MakeCode Arcade with SAMD51 M4](https://learn.adafruit.com/makecode-arcade-with-samd51-m4.md)
- [CircuitPython 101: Functions](https://learn.adafruit.com/circuitpython-101-functions.md)
- [Custom Color Palettes for MakeCode Arcade Games](https://learn.adafruit.com/custom-color-palettes-for-makecode-arcade-games.md)
- [IoT Air Quality Sensor with Adafruit IO](https://learn.adafruit.com/diy-air-quality-monitor.md)
- [CircuitPython Turtle Graphics](https://learn.adafruit.com/circuitpython-turtle-graphics.md)
- [Program CircuitPython USB Devices with iPhone & iPad](https://learn.adafruit.com/use-circuitpython-devices-with-iphone-ipad.md)
- [CircuitPython 101: Basic Builtin Data Structures](https://learn.adafruit.com/basic-datastructures-in-circuitpython.md)
- [Playing Arduboy Games on Arcada](https://learn.adafruit.com/playing-arduboy-games-on-arcada.md)
- [Adding a WiFi Co-Processor to CircuitPython](https://learn.adafruit.com/adding-a-wifi-co-processor-to-circuitpython-esp8266-esp32.md)
- [CircuitPython 101: Working with Lists, Iterators and Generators](https://learn.adafruit.com/circuitpython-101-list-and-things-iterators-generators.md)
- [Esenciales para CircuitPython](https://learn.adafruit.com/esenciales-para-circuitpython.md)
- [Use circup to easily keep your CircuitPython libraries up to date](https://learn.adafruit.com/keep-your-circuitpython-libraries-on-devices-up-to-date-with-circup.md)
- [IoT Moon Phase Guide](https://learn.adafruit.com/moon-phase.md)
- [Adafruit MagTag Project Selector](https://learn.adafruit.com/adafruit-magtag-project-selector.md)