The core idea of this project is to give an LLM agent access to a device that can be used to sense the environment that its human user is in, and to express itself in that environment using sensor feedback loops to confirm the physical effects. This version of the project uses an Adafruit ESP32-S3 Reverse TFT Feather as the central hardware.
This quote from the Abstract section of the inspiration paper sums it up nicely.
We present a minimal hardware-software architecture that grants a large language model a closed-loop physical embodiment: six input modalities (temperature, humidity, atmospheric pressure, illuminance, motion, sound) across four sensor modules, three output channels (haptic, visual, audio), and two input-output couplings that let the LLM verify its outputs land in the physical world. The system runs on a single microcontroller exposed as a network-accessible API; a remote LLM client perceives its surroundings, expresses into them, and receives back — via paired on-board sensors — confirmation of its own outputs in two modalities (audio via microphone, haptic via accelerometer), constituting self-perception loops. We identify this three-part structure — perception, expression, and self-perception of expression — as a minimal sufficient configuration for closed-loop physical agency — the capacity to act in a physical environment and perceive the consequence — in an LLM.
With the embodiment kit an agent can sense:
- temperature, humidity, and pressure with a BME280
- sound level with a PDM mic
- light with a VEML770
- motion with an LIS3DH
- button presses with 3 built-in buttons
It can output effects with the following:
- sound with a piezo buzzer
- RGB light with NeoPixels
- haptic vibration with a DRV2605 + haptic motor
- TFT display to show faces or text
The buzzer, NeoPixels, and haptic motor are all tied to sensor feedback loops to enable confirmation of their effect. Buzzer with PDM mic, NeoPixels with light sensor, and haptic motor with accelerometer.
Adaptability
The hardware and software is built to be modular and adaptable to different needs and use-cases. The embodiment kit was primarily intended for use with an LLM agent, but its CLI script can easily be used from a terminal or integrated in other non-LLM contexts as well. This guide documents building the kit with all of the hardware components listed above, but the code is easy to adapt to different or fewer sensors.
New Territory
This project differs from the original version by Olivia Zhu in a number of ways. It's implemented in CircuitPython instead of Arduino. Communication between embodiment device and agent occurs over Adafruit IO MQTT, or on the local network with a small HTTP server running on the microcontroller itself. A larger bridge server is not necessary. A NeoPixel stick has been added and paired with the lux sensor for an additional feedback loop.
The TFT display is a bit larger and can handle color. A set of cute facial expression sprites are used instead of programmatic drawing. The display can be used to show text in addition to graphics. The push buttons built-in to the Feather can be used to prompt the user with up to 3 options and receive feedback about their selection. Different specific sensors and breakouts are used, but they can sense the same things. STEMMA QT cables are used for all I2C connections.
Page last edited May 26, 2026
Text editor powered by tinymce.
