Build a machine learning (ML) package detector that sends you a notification when a package is left at your door!

You can also use this tutorial to train a model to notify you for any kind of event. For example, lets you know when your dog is at the door, your favorite parking spot is open, or if there are birds at your bird feeder.

We'll use Lobe to train our model and then deploy it to a Raspberry Pi 4. We'll use the BrainCraft HAT to interface with the Pi to collect images and preview what the camera is seeing. Next, we'll setup an applet on If This Then That to send us an e-mail alert whenever a package is detected. Finally, we'll improve our model by collecting more training data in situations where the model gets confused.

This tutorial is part of a series which includes:

Background Knowledge

New to Lobe? At a minimum, we recommend following the introductory Tutorial 1 above. 

To be successful with this project, you'll need some experience with the following:

  1. Setting up and using the Raspberry Pi
  2. Some familiarity with using the terminal window
  3. Installing the Pi Camera
Machine learning is a transformative tool that’s redefining how we build software—but up until now, it was only accessible to a small group of experts. At Adafruit, we...
Out of Stock

Hardware Setup

Before starting, set up your Pi and BrainCraft HAT. Follow this primary guide for the BrainCraft to configure these services:

  • Blinka
  • Fan Service
  • Display Module

Installing Lobe SDK

Connect to your Pi using SSH and run the following script to install the Lobe SDK:

cd ~
wget https://raw.githubusercontent.com/lobe/lobe-python/master/scripts/lobe-rpi-install.sh
sudo bash lobe-rpi-install.sh

Note: To setup SSH, follow this guide.

Download the GitHub repo into the Pi's home folder with the following commands:

cd ~
git clone https://github.com/lobe/lobe-adafruit-kit.git

The GitHub repo can also be found here.

Since we're using the Pi in a headless configuration, we'll use an FTP connection to transfer files between our computer and the Pi.

Windows Instructions

Download and install WinSCP

 

Open WinSCP and start a New Session

Select an SFTP connection, fill in the IP address of your Pi, set the username to Pi, and put in your password.

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Mac Instructions

Download and install FileZilla. When it's done installing, open the program.

Type sftp:// followed by the IP address of your Pi. Set the username to pi and put in your password.

Your Pi's IP address is on the screen of the BrainCraft. You can also use the hostname address, e.g. "raspberrypi.local" ([email protected]).

Before we can train an ML model, we need to collect some images. For this model, we'll need two categories: images that have a package in them and images that don't.

To make our model more accurate under various weather conditions and other situations, it's useful to have lots of pictures in both categories at different times of day and in different lighting conditions.

Install your Pi with the Camera pointing towards a package drop-off spot.

I installed my Pi just inside the door frame, with the camera outside and pointing down.

Open a terminal window and connect to your Pi via SSH.

Tip: Your IP address is displayed on the screen of the BrainCraft

In terminal, run the lobe-capture.py program with the following commands:

cd ~
cd lobe-adafruit-kit
python3 lobe-capture.py

Collect training images

Using the button on the BrainCraft, take 20-30 pictures with no packages. These pictures will be our baseline.

Holding the button down will take a burst of pictures.

Next take 20-30 with packages.

Use as many different boxes as you have, and try moving the packages around, stacking them, and using a different number.

Upload all the pictures to your computer using FTP, and put them in a folder called Package Detector.

Inside of that folder create two more folders called no package and package. Sort your images into those two folders.

Make sure your folder names are the same (including punctuation) for the sample code to work!

Open Lobe and create a new project.

Select Import in the top left, then select Dataset from the drop-down menu.

Choose the Package Detector folder you created in the previous step. Lobe will automatically start training your model!

You may need to optimize your model before exporting it. This allows Lobe to train for longer and continue to find improvements for your model. To optimize, click the menu icon and select "Optimize Model".

You can do a basic test of your model on your computer, but it is best to test your model in the location it will be used, like your front door.

To fully test the model, we'll have to deploy it to the Pi. Onward!

Next, export your Lobe model to use on the Raspberry Pi. We'll use TensorFlow Lite which is a format that is optimized for mobile and edge devices, like the Pi. 

In Lobe, navigate to the Use tab and click Export.

Select TensorFlow Lite and select a location to save the model. We'll transfer the model to our Raspberry Pi later in the tutorial.

Uploading the model to the Pi

In terminal, connect to your Pi using SSH and create a directory named model via the following commands:

cd ~
mkdir model
The IP address is on the screen of your BrainCraft Hat

Open an FTP connection and transfer the model.tflite and signature.json files exported from Lobe into the model directory on the Pi.

Run the Package Detector Code

Run the package detector code with the following commands:

cd ~
cd lobe-adafruit-kit
python3 lobe-package-detector.py

Next, go look at your Pi! The BrainCraft screen should now show you what the camera is seeing along with a prediction label.

Try putting a package (or an empty box) in the camera field-of-view and see if the Pi detects the package.

If the model does not recognize the package, add more training images to your model.

Now that your Pi is making prediction on package deliveries, let's set up some alerts!

Set up an IFTTT Applet

Go to IFTTT and log in (or create an account).

Once you are logged in, click Create in the top right corner to create a new Applet.

Click the Add button on the If This block.

Search for and select Webhooks.

Set the Event Name trigger to package, and click on Create trigger.

This will trigger an action when the ML model label is "package". 

If you are using a different ML model label than "package", be sure to change the Event Name trigger!

Next, click the Add button on the Then That block.

Search for and select Email.

If this is the first time you're setting up an e-mail applet in IFTTT, you'll need to verify your e-mail.

Edit the subject line and body text of the email notification. This is the email you'll see for each package notification. The items highlighted in gray are dynamic elements that are generated when the event runs.

Finally give your applet a name and click Finish.

Your email notification applet is now ready to go!

Setup Webhook Key

We're using a webhook to trigger the email notification.  The webhook has a unique key, so only you can trigger the event. We're going to get the key from IFTTT and set it as an environment variable on the Pi. 

From your profile menu in the top right, select My services.

Next, select Webhooks.

From the Webhooks screen, select Documentation in the top right.

Copy the key from the top of the page.

Remember to treat the key like a password

Open an SSH connection to the Pi. Create a new environment variable called IFTTTKEY.

Use the following command in terminal, replacing PASTE-KEY-HERE with your IFTTT key:

export IFTTTKEY=PASTE-KEY-HERE

Start the program again

Finally, restart the lobe-package-detector.py program and you're all set to go!

You may notice that your package detector is sending you notifications when things that aren't packages are in its view (e.g. people, birds, etc.).

To improve our model, we'll use continuous learning by taking pictures of things that confuse our model, and then retrain the model using these new images.

Collect some other objects that aren't packages that might be by your front door. Wait for the model to make a prediction, then push and hold the joy stick up (towards the LEDs) on the BrainCraft if the label you see is correct, the LEDs will blink green when the picture is saved. If the label is wrong, push and hold the joy stick down (away from the LEDs) the LEDs will blink red when the picture is saved. 

This will save the images into a structed dataset folder named retraining_data.

Take 10-20 pictures of each object using the joystick on the Pi. 

Open an FTP Connection to the Pi and copy the retraining_data folder to your computer.

Open the Package Detector project in Lobe.

Import the structured dataset you created on your Pi.

Any images saved in the top level directory, retraining_data in this case, will need to be labeled in Lobe. You can put these photos in the "no package" category or, if you want a more detailed package detector, you can create new categories!

When you're done labeling, export your updated model and deploy it to the Pi. Follow instructions in "Export your Model". 

Now that your model is trained and improved, and you're getting email notifications from just a package, there are lots of other ways you can extend this project!

You might want to use this notify you when your pet is at the door or if there is food being delivered. All you have to do is add a new label in Lobe, add another trigger in IFTTT, and you'll have a different alert!

This guide was first published on Mar 30, 2021. It was last updated on 2021-03-30 15:53:33 -0400.