# Build an ML Package Detector with Lobe ## Overview Build a machine learning (ML) package detector that sends you a notification when a package is left at your door! You can also use this tutorial to train a model to notify you for any kind of event. For example, lets you know when your dog is at the door, your favorite parking spot is open, or if there are birds at your bird feeder. We'll use Lobe to train our model and then deploy it to a Raspberry Pi 4. We'll use the BrainCraft HAT to interface with the Pi to collect images and preview what the camera is seeing. Next, we'll setup an applet on If This Then That to send us an e-mail alert whenever a package is detected. Finally, we'll improve our model by collecting more training data in situations where the model gets confused. This tutorial is part of a series which includes: - [Tutorial 1: Machine Learning 101 with Lobe and Raspberry Pi](https://learn.adafruit.com/run-lobe-on-braincraft/) - [Tutorial 2: Build an ML Rock Paper Scissors Game with Lobe](https://learn.adafruit.com/lobe-rock-paper-scissors) ## Background Knowledge New to Lobe? At a minimum, we recommend following the introductory Tutorial 1 above.  To be successful with this project, you'll need some experience with the following: 1. Setting up and using the Raspberry Pi 2. Some familiarity with using the terminal window 3. Installing the Pi Camera # Build an ML Package Detector with Lobe ## Required Parts # Build an ML Package Detector with Lobe ## Set up your Pi ## Hardware Setup Before starting, set up your Pi and BrainCraft HAT. Follow [this primary guide for the BrainCraft](https://learn.adafruit.com/lobe-braincraft/raspberry-pi-setup) to configure these services: - Blinka - Fan Service - Display Module ## Installing Lobe SDK Connect to your Pi using SSH and run the following script to install the Lobe SDK: ```none cd ~ wget https://raw.githubusercontent.com/lobe/lobe-python/master/scripts/lobe-rpi-install.sh sudo bash lobe-rpi-install.sh ``` _Note: To setup SSH, follow [this guide](https://learn.adafruit.com/adafruits-raspberry-pi-lesson-6-using-ssh)._ # Build an ML Package Detector with Lobe ## Download the Code Download the GitHub repo into the Pi's home folder with the following commands: ```none cd ~ git clone https://github.com/lobe/lobe-adafruit-kit.git ``` The GitHub repo can also be found [here](https://github.com/lobe/lobe-adafruit-kit). # Build an ML Package Detector with Lobe ## Set up an FTP Connection Since we're using the Pi in a headless configuration, we'll use an FTP connection to transfer files between our computer and the Pi. ## Windows Instructions Download and install [WinSCP](https://winscp.net/eng/index.php)   ![](https://cdn-learn.adafruit.com/assets/assets/000/099/376/medium800/adafruit_products_winscp-new-session.png?1613066405) Open WinSCP and start a New Session ![](https://cdn-learn.adafruit.com/assets/assets/000/099/377/medium800/adafruit_products_winscp-new-session-details.png?1613066416) Select an SFTP connection, fill in the IP address of your Pi, set the username to Pi, and put in your password. Info: ## Mac Instructions Download and install [FileZilla](https://filezilla-project.org/download.php?type=client). When it's done installing, open the program. ![](https://cdn-learn.adafruit.com/assets/assets/000/101/090/medium800/adafruit_products_FileZilla_config.png?1617126693) Type `sftp://` followed by the IP address of your Pi. Set the username to `pi` and put in your password. Info: ![](https://cdn-learn.adafruit.com/assets/assets/000/101/091/medium800/adafruit_products_FileZilla_connected.png?1617126721) # Build an ML Package Detector with Lobe ## Collect Images Before we can train an ML model, we need to collect some images. For this model, we'll need two categories: images that have a package in them and images that don't. To make our model more accurate under various weather conditions and other situations, it's useful to have lots of pictures in both categories at different times of day and in different lighting conditions. ## Install your Pi with the Camera pointing towards a package drop-off spot. I installed my Pi just inside the door frame, with the camera outside and pointing down. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/813/medium800/camera_20210316_195727082_iOS.jpg?1616186528) ![](https://cdn-learn.adafruit.com/assets/assets/000/100/814/medium800/camera_20210316_195741899_iOS.jpg?1616186542) ![](https://cdn-learn.adafruit.com/assets/assets/000/100/739/medium800/camera_ssh.png?1615924768) Open a terminal window and connect to your Pi via SSH. Info: In terminal, run the `lobe-capture.py` program with the following commands: ```none cd ~ cd lobe-adafruit-kit python3 lobe-capture.py ``` ## Collect training images Using the button on the BrainCraft, take 20-30 pictures with **no packages**. These pictures will be our baseline. Holding the button down will take a burst of pictures. ![camera_0.jpg](https://cdn-learn.adafruit.com/assets/assets/000/100/752/medium640/camera_0.jpg?1615939770) Next take 20-30 **with packages.** ![camera_41.jpg](https://cdn-learn.adafruit.com/assets/assets/000/100/753/medium640/camera_41.jpg?1615939793) Info: ![](https://cdn-learn.adafruit.com/assets/assets/000/100/961/medium800/camera_dataset_folder.png?1616617553) Upload all the pictures to your computer using FTP, and put them in a folder called `Package Detector`. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/962/medium800/camera_sorted.png?1616617561) Inside of that folder **create two more folders** called `no package` and `package`. Sort your images into those two folders. Info: # Build an ML Package Detector with Lobe ## Train your Model Open Lobe and create a new project. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/774/medium800/camera_import_dataset.png?1616011030) Select Import in the top left, then select Dataset from the drop-down menu. ![](https://cdn-learn.adafruit.com/assets/assets/000/101/077/medium800/camera_Tutorial3-ImportPackageDetectorDataset.png?1617064889) Choose the `Package Detector` folder you created in the previous step. Lobe will automatically start training your model! ![](https://cdn-learn.adafruit.com/assets/assets/000/100/776/medium800/camera_optimize.png?1616011564) You may need to optimize your model before exporting it. This allows Lobe to train for longer and continue to find improvements for your model. To optimize, click the menu icon and select "Optimize Model". You can do a basic test of your model on your computer, but it is best to test your model in the location it will be used, like your front door. To fully test the model, we'll have to deploy it to the Pi. Onward! # Build an ML Package Detector with Lobe ## Export your Model Next, export your Lobe model to use on the Raspberry Pi. We'll use **TensorFlow Lite** which is a format that is optimized for mobile and edge devices, like the Pi.  ![](https://cdn-learn.adafruit.com/assets/assets/000/101/098/medium800/adafruit_products_6._Export.png?1617128073) In Lobe, navigate to the Use tab and click Export. Select TensorFlow Lite and select a location to save the model. We'll transfer the model to our Raspberry Pi later in the tutorial. # Build an ML Package Detector with Lobe ## Get Predictions on the Pi ## Uploading the model to the Pi In terminal, connect to your Pi using SSH and create a directory named `model` via the following commands: ```none cd ~ mkdir model ``` Info: ![](https://cdn-learn.adafruit.com/assets/assets/000/100/812/medium800/camera_move_model.png?1616186232) Open an FTP connection and transfer the `model.tflite` and `signature.json` files exported from Lobe into the `model` directory on the Pi. ## Run the Package Detector Code Run the package detector code with the following commands: ```none cd ~ cd lobe-adafruit-kit python3 lobe-package-detector.py ``` Next, go look at your Pi! The BrainCraft screen should now show you what the camera is seeing along with a prediction label. Try putting a package (or an empty box) in the camera field-of-view and see if the Pi detects the package. If the model does not recognize the package, add more training images to your model. # Build an ML Package Detector with Lobe ## Set up E-mail Alerts Now that your Pi is making prediction on package deliveries, let's set up some alerts! ## Set up an IFTTT Applet Go to [IFTTT](https://www.ifttt.com) and log in (or create an account). ![](https://cdn-learn.adafruit.com/assets/assets/000/100/780/medium800/camera_ifcreate.png?1616084886) Once you are logged in, click **_Create_** in the top right corner to create a new Applet. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/781/medium800/camera_thisadd.png?1616084978) Click the _ **Add** _ button on the _If This_ block. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/783/medium800/camera_webhooks.png?1616088170) Search for and select _ **Webhooks.** _ ![](https://cdn-learn.adafruit.com/assets/assets/000/100/784/medium800/camera_webhook_event.png?1616088789) Set the Event Name trigger to _ **package** _, and click on _ **Create trigger** _. This will trigger an action when the ML model label is "package".  Info: ![](https://cdn-learn.adafruit.com/assets/assets/000/100/785/medium800/camera_thatadd.png?1616090644) Next, click the _ **Add** _ button on the _Then That_ block. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/789/medium800/camera_email.png?1616104134) Search for and select _ **Email.** _ ![](https://cdn-learn.adafruit.com/assets/assets/000/100/790/medium800/camera_connectemail.png?1616104194) If this is the first time you're setting up an e-mail applet in IFTTT, you'll need to verify your e-mail. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/791/medium800/camera_createemailbody.png?1616104203) Edit the subject line and body text of the email notification. This is the email you'll see for each package notification. The items highlighted in gray are dynamic elements that are generated when the event runs. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/796/medium800/camera_appletname.png?1616110010) Finally give your applet a name and click _ **Finish** _. Your email notification applet is now ready to go! ## Setup Webhook Key We're using a webhook to trigger the email notification.  The webhook has a unique key, so only you can trigger the event. We're going to get the key from IFTTT and set it as an environment variable on the Pi.  ![](https://cdn-learn.adafruit.com/assets/assets/000/100/792/medium800/camera_myservices.png?1616105371) From your profile menu in the top right, select _ **My services.** _ ![](https://cdn-learn.adafruit.com/assets/assets/000/100/793/medium800/camera_webhook_setting.png?1616105764) Next, select _ **Webhooks.** _ ![](https://cdn-learn.adafruit.com/assets/assets/000/100/794/medium800/camera_webhook_docs.png?1616105786) From the Webhooks screen, select _ **Documentation** _ in the top right. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/795/medium800/camera_webhook_key.png?1616105822) Copy the key from the top of the page. Danger: ![](https://cdn-learn.adafruit.com/assets/assets/000/100/963/medium800/camera_setenv.png?1616617664) Open an SSH connection to the Pi. Create a new environment variable called `IFTTTKEY`. Use the following command in terminal, replacing `PASTE-KEY-HERE` with your IFTTT key: ```none export IFTTTKEY=PASTE-KEY-HERE ``` ## Start the program again Finally, restart the `lobe-package-detector.py` program and you're all set to go! # Build an ML Package Detector with Lobe ## Continuous Learning You may notice that your package detector is sending you notifications when things that aren't packages are in its view (e.g. people, birds, etc.). To improve our model, we'll use **continuous learning** by taking pictures of things that confuse our model, and then retrain the model using these new images. Collect some other objects that aren't packages that might be by your front door. Wait for the model to make a prediction, then push and hold the joy stick up (towards the LEDs) on the BrainCraft if the label you see is correct, the LEDs will blink green when the picture is saved. If the label is wrong, push and hold the joy stick down (away from the LEDs) the LEDs will blink red when the picture is saved.  This will save the images into a structed dataset folder named `retraining_data`. ![](https://cdn-learn.adafruit.com/assets/assets/000/101/076/medium800/camera_Tutorial3-TestDeployedModel.jpg?1617063360) Take 10-20 pictures of each object using the joystick on the Pi.  ![](https://cdn-learn.adafruit.com/assets/assets/000/100/806/medium800/camera_retrainingdata.png?1616173525) Open an FTP Connection to the Pi and copy the `retraining_data` folder to your computer. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/805/medium800/camera_loadproject.png?1616173269) Open the Package Detector project in Lobe. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/804/medium800/camera_import2.png?1616173181) Import the structured dataset you created on your Pi. ![](https://cdn-learn.adafruit.com/assets/assets/000/100/803/medium800/camera_label-retrain.png?1616173167) Any images saved in the top level directory, `retraining_data` in this case, will need to be labeled in Lobe. You can put these photos in the "no package" category or, if you want a more detailed package detector, you can create new categories! When you're done labeling, export your updated model and deploy it to the Pi. Follow instructions in "[Export your Model](https://learn.adafruit.com/build-an-ml-package-detector/export-your-model)".  # Build an ML Package Detector with Lobe ## Going Further Now that your model is trained and improved, and you're getting email notifications from just a package, there are lots of other ways you can extend this project! You might want to use this notify you when your pet is at the door or if there is food being delivered. All you have to do is add a new label in Lobe, add another trigger in IFTTT, and you'll have a different alert! ## Related Guides - [Raspberry Pi Care and Troubleshooting](https://learn.adafruit.com/raspberry-pi-care-and-troubleshooting.md) - [Adafruit BrainCraft HAT - Easy Machine Learning for Raspberry Pi](https://learn.adafruit.com/adafruit-braincraft-hat-easy-machine-learning-for-raspberry-pi.md) - [Machine Learning 101 with Lobe and BrainCraft](https://learn.adafruit.com/machine-learning-101-lobe-braincraft.md) - [Raspberry Pi HQ Camera Case](https://learn.adafruit.com/raspberry-pi-hq-camera-case.md) - [Raspberry Pi E-Ink Weather Station using Python](https://learn.adafruit.com/raspberry-pi-e-ink-weather-station-using-python.md) - [Running PyPortal Code on Blinka with Displayio](https://learn.adafruit.com/running-pyportal-code-on-blinka-with-displayio.md) - [Video Feedback with Raspberry Pi](https://learn.adafruit.com/video-feedback-with-raspberry-pi.md) - [Automatic Naughty Cat Detector using Lobe](https://learn.adafruit.com/naughty-cat-detector-using-microsoft-lobe.md) - [Raspberry Pi YouTube Boombox](https://learn.adafruit.com/youtube-radio.md) - [Raspberry Pi E-Ink Event Calendar using Python](https://learn.adafruit.com/raspberry-pi-e-ink-desk-calendar-using-python.md) - [Using Google Assistant on the BrainCraft HAT or Voice Bonnet](https://learn.adafruit.com/using-google-assistant-on-the-braincraft-hat.md) - [Programming with Scratch 2 or 3 on Raspberry Pi](https://learn.adafruit.com/programming-with-scratch-on-raspberry-pi.md) - [Raspberry Pi Azure IoT Hub Dashboard with CircuitPython](https://learn.adafruit.com/raspberry-pi-iot-dashboard-with-azure-and-circuitpython.md) - [Raspberry Pi Video Synth with Blinka and Processing](https://learn.adafruit.com/raspberry-pi-video-synth-with-blinka-and-processing.md) - [Running TensorFlow Lite Object Recognition on the Raspberry Pi 4 or Pi 5](https://learn.adafruit.com/running-tensorflow-lite-on-the-raspberry-pi-4.md) - [Set up Home Assistant with a Raspberry Pi](https://learn.adafruit.com/set-up-home-assistant-with-a-raspberry-pi.md) - [Raspberry Pi Pedal for Youtube](https://learn.adafruit.com/raspberry-pi-pedal-for-youtube.md) - [Visual Studio Code for the Raspberry Pi](https://learn.adafruit.com/visual-studio-code-for-the-raspberry-pi.md)