Build a machine learning (ML) package detector that sends you a notification when a package is left at your door!

You can also use this tutorial to train a model to notify you for any kind of event. For example, lets you know when your dog is at the door, your favorite parking spot is open, or if there are birds at your bird feeder.

We'll use Lobe to train our model and then deploy it to a Raspberry Pi 4. We'll use the BrainCraft HAT to interface with the Pi to collect images and preview what the camera is seeing. Next, we'll setup an applet on If This Then That to send us an e-mail alert whenever a package is detected. Finally, we'll improve our model by collecting more training data in situations where the model gets confused.

This tutorial is part of a series which includes:

• Tutorial 1: Machine Learning 101 with Lobe and Raspberry Pi
• Tutorial 2: Build an ML Rock Paper Scissors Game with Lobe

Background Knowledge

New to Lobe? At a minimum, we recommend following the introductory Tutorial 1 above. 

To be successful with this project, you'll need some experience with the following:

1. Setting up and using the Raspberry Pi
2. Some familiarity with using the terminal window
3. Installing the Pi Camera

Before we can train an ML model, we need to collect some images. For this model, we'll need two categories: images that have a package in them and images that don't.

To make our model more accurate under various weather conditions and other situations, it's useful to have lots of pictures in both categories at different times of day and in different lighting conditions.

Install your Pi with the Camera pointing towards a package drop-off spot.

I installed my Pi just inside the door frame, with the camera outside and pointing down.

Open a terminal window and connect to your Pi via SSH.

In terminal, run the lobe-capture.py program with the following commands:

Collect training images

Upload all the pictures to your computer using FTP, and put them in a folder called Package Detector.

Inside of that folder create two more folders called no package and package. Sort your images into those two folders.

Open Lobe and create a new project.

Select Import in the top left, then select Dataset from the drop-down menu.

Choose the Package Detector folder you created in the previous step. Lobe will automatically start training your model!

You may need to optimize your model before exporting it. This allows Lobe to train for longer and continue to find improvements for your model. To optimize, click the menu icon and select "Optimize Model".

You can do a basic test of your model on your computer, but it is best to test your model in the location it will be used, like your front door.

To fully test the model, we'll have to deploy it to the Pi. Onward!

Uploading the model to the Pi

In terminal, connect to your Pi using SSH and create a directory named model via the following commands:

Open an FTP connection and transfer the model.tflite and signature.json files exported from Lobe into the model directory on the Pi.

Run the Package Detector Code

Run the package detector code with the following commands:

Next, go look at your Pi! The BrainCraft screen should now show you what the camera is seeing along with a prediction label.

Try putting a package (or an empty box) in the camera field-of-view and see if the Pi detects the package.

If the model does not recognize the package, add more training images to your model.

Now that your Pi is making prediction on package deliveries, let's set up some alerts!

Set up an IFTTT Applet

Go to IFTTT and log in (or create an account).

Once you are logged in, click Create in the top right corner to create a new Applet.

Click the Add button on the If This block.

Search for and select Webhooks.

Set the Event Name trigger to package, and click on Create trigger.

This will trigger an action when the ML model label is "package". 

Next, click the Add button on the Then That block.

Search for and select Email.

If this is the first time you're setting up an e-mail applet in IFTTT, you'll need to verify your e-mail.

Edit the subject line and body text of the email notification. This is the email you'll see for each package notification. The items highlighted in gray are dynamic elements that are generated when the event runs.

Finally give your applet a name and click Finish.

Your email notification applet is now ready to go!

Setup Webhook Key

We're using a webhook to trigger the email notification.  The webhook has a unique key, so only you can trigger the event. We're going to get the key from IFTTT and set it as an environment variable on the Pi. 

From your profile menu in the top right, select My services.

Next, select Webhooks.

From the Webhooks screen, select Documentation in the top right.

Copy the key from the top of the page.

Open an SSH connection to the Pi. Create a new environment variable called IFTTTKEY.

Use the following command in terminal, replacing PASTE-KEY-HERE with your IFTTT key:

Start the program again

Finally, restart the lobe-package-detector.py program and you're all set to go!

You may notice that your package detector is sending you notifications when things that aren't packages are in its view (e.g. people, birds, etc.).

To improve our model, we'll use continuous learning by taking pictures of things that confuse our model, and then retrain the model using these new images.

Collect some other objects that aren't packages that might be by your front door. Wait for the model to make a prediction, then push and hold the joy stick up (towards the LEDs) on the BrainCraft if the label you see is correct, the LEDs will blink green when the picture is saved. If the label is wrong, push and hold the joy stick down (away from the LEDs) the LEDs will blink red when the picture is saved. 

This will save the images into a structed dataset folder named retraining_data.

Take 10-20 pictures of each object using the joystick on the Pi. 

Open an FTP Connection to the Pi and copy the retraining_data folder to your computer.

Open the Package Detector project in Lobe.

Import the structured dataset you created on your Pi.

Any images saved in the top level directory, retraining_data in this case, will need to be labeled in Lobe. You can put these photos in the "no package" category or, if you want a more detailed package detector, you can create new categories!

When you're done labeling, export your updated model and deploy it to the Pi. Follow instructions in "Export your Model". 

Now that your model is trained and improved, and you're getting email notifications from just a package, there are lots of other ways you can extend this project!

You might want to use this notify you when your pet is at the door or if there is food being delivered. All you have to do is add a new label in Lobe, add another trigger in IFTTT, and you'll have a different alert!