OpenAI Image Descriptors with MEMENTO

by Liz Clark

published March 04, 2024, last edited March 04, 2024
last major update March 04, 2024

posted in Sensors/ Camera Internet of Things - IOT/ Wireless CircuitPython Programming/ MicroPython / CircuitPython

Intermediate

Project guide

Configuring settings.toml and the OpenAI Prompts

This project depends on you adding your WiFi settings, OpenAI API key and text prompts to your board's settings.toml file. The text prompts are available in a text file in the Project Bundle.

Download File

Copy Code

# SPDX-FileCopyrightText: 2024 Liz Clark for Adafruit Industries
#
# SPDX-License-Identifier: MIT

ALT_TEXT_PROMPT="Please describe what is in this image. The description should be in the style of alt text used on social media sites like Mastodon. Avoid talking about the quality of the image. Avoid talking about the background of the image. Only describe the main subject."
HAIKU_PROMPT="Please identify the main subject of the image and write a haiku about it. It should be a three line poem that consists of three lines, with five syllables in the first line, seven in the second, and five in the third. It does not need to rhyme. Only return the haiku, no image description is necessary. Please insert a * between each line instead of a comma (,)."
CABLE_PROMPT="This image features a type of cable connector. Please specifically identify the cable connection. You should describe the category of the cable and the specific type. For example, if you see a USB cable, you should also say whether it is type A, type mini B, etc. If you are not sure what type of cable connector it is, please state that and make your best inference. The response should be concise, no more than two sentences. You do not need to describe the general appearance of the image."
TRANSLATE_PROMPT="This image has text that is not in English. Please translate it to English. Only return the translated text in English. For example, if the text reads hola (hello in Spanish), then return hello (translated to English). No further explanation is necessary. If you are not sure on how to translate a word(s) then please return the text as seen in the image. You do not need to describe the image."
ALIEN_PROMPT="Describe this image as if you are a friendly alien visiting from another world in our galaxy. You have no context for society on Earth. Your descriptions are cute, inquisitive and concise (2-3 sentences). You do not need to comment on the quality of the image."
WEIRD_PROMPT="Take in the full image as context. Is there anything in the image that is out of the ordinary or interesting? For example, if the image is of a backyard, do you see any animals or objects that catch your eye? You do not need to comment on the quality of the image. Please keep responses brief and concise, between one to two sentences. If you do not see anything out of the ordinary for the context of the image or anything particularly interesting, please return: Nothing to see here."

# SPDX-FileCopyrightText: 2024 Liz Clark for Adafruit Industries
#
# SPDX-License-Identifier: MIT

ALT_TEXT_PROMPT="Please describe what is in this image. The description should be in the style of alt text used on social media sites like Mastodon. Avoid talking about the quality of the image. Avoid talking about the background of the image. Only describe the main subject."
HAIKU_PROMPT="Please identify the main subject of the image and write a haiku about it. It should be a three line poem that consists of three lines, with five syllables in the first line, seven in the second, and five in the third. It does not need to rhyme. Only return the haiku, no image description is necessary. Please insert a * between each line instead of a comma (,)."
CABLE_PROMPT="This image features a type of cable connector. Please specifically identify the cable connection. You should describe the category of the cable and the specific type. For example, if you see a USB cable, you should also say whether it is type A, type mini B, etc. If you are not sure what type of cable connector it is, please state that and make your best inference. The response should be concise, no more than two sentences. You do not need to describe the general appearance of the image."
TRANSLATE_PROMPT="This image has text that is not in English. Please translate it to English. Only return the translated text in English. For example, if the text reads hola (hello in Spanish), then return hello (translated to English). No further explanation is necessary. If you are not sure on how to translate a word(s) then please return the text as seen in the image. You do not need to describe the image."
ALIEN_PROMPT="Describe this image as if you are a friendly alien visiting from another world in our galaxy. You have no context for society on Earth. Your descriptions are cute, inquisitive and concise (2-3 sentences). You do not need to comment on the quality of the image."
WEIRD_PROMPT="Take in the full image as context. Is there anything in the image that is out of the ordinary or interesting? For example, if the image is of a backyard, do you see any animals or objects that catch your eye? You do not need to comment on the quality of the image. Please keep responses brief and concise, between one to two sentences. If you do not see anything out of the ordinary for the context of the image or anything particularly interesting, please return: Nothing to see here."

Plug your CircuitPython board into your computer via a known good data + power USB cable. Your board should show up as a thumb drive in your File Explorer / Finder (depending on your operating system) named CIRCUITPY.

Edit the settings.toml file to contain the following keys:

CIRCUITPY_WIFI_SSID
CIRCUITPY_WIFI_PASSWORD
OPENAI_API_KEY
ALT_TEXT_PROMPT
HAIKU_PROMPT
CABLE_PROMPT
TRANSLATE_PROMPT
ALIEN_PROMPT
WEIRD_PROMPT

Your file should look similar to the one shown below:

Download File

Copy Code

CIRCUITPY_WIFI_SSID = "YOUR-SSID-HERE"
CIRCUITPY_WIFI_PASSWORD = "YOUR-PASSWORD-HERE"
OPENAI_API_KEY="YOUR-OPENAI-KEY-HERE"
ALT_TEXT_PROMPT="Please describe what is in this image. The description should be in the style of alt text used on social media sites like Mastodon. Avoid talking about the quality of the image. Avoid talking about the background of the image. Only describe the main subject."
HAIKU_PROMPT="Please identify the main subject of the image and write a haiku about it. It should be a poem that consists of three lines, with five syllables in the first line, seven in the second, and five in the third. It does not need to rhyme. Only return the haiku, no image description is necessary. Please insert a * between each line instead of a comma (,)."
CABLE_PROMPT="This image features a type of cable connector. Please specifically identify the cable connection. You should describe the category of the cable and the specific type. For example, if you see a USB cable, you should also say whether it is type A, type mini B, etc. If you are not sure what type of cable connector it is, please state that and make your best inference. The response should be concise, no more than two sentences. You do not need to describe the general appearance of the image."
TRANSLATE_PROMPT="This image has text that is not in English. Please translate it to English. Only return the translated text in English. For example, if the text reads hola (hello in Spanish), then return hello (translated to English). No further explanation is necessary. If you are not sure on how to translate a word(s) then please return the text as seen in the image. You do not need to describe the image."
ALIEN_PROMPT="Describe this image as if you are a friendly alien visiting from another world in our galaxy. You have no context for society on Earth. Your descriptions are cute, inquisitive and concise (2-3 sentences). You do not need to comment on the quality of the image."
WEIRD_PROMPT="Take in the full image as context. Is there anything in the image that is out of the ordinary or interesting? For example, if the image is of a backyard, do you see any animals or objects that catch your eye? You do not need to comment on the quality of the image. Please keep responses brief and concise, between one to two sentences. If you do not see anything out of the ordinary for the context of the image or anything particularly interesting, please return: Nothing to see here."

CIRCUITPY_WIFI_SSID = "YOUR-SSID-HERE"
CIRCUITPY_WIFI_PASSWORD = "YOUR-PASSWORD-HERE"
OPENAI_API_KEY="YOUR-OPENAI-KEY-HERE"
ALT_TEXT_PROMPT="Please describe what is in this image. The description should be in the style of alt text used on social media sites like Mastodon. Avoid talking about the quality of the image. Avoid talking about the background of the image. Only describe the main subject."
HAIKU_PROMPT="Please identify the main subject of the image and write a haiku about it. It should be a poem that consists of three lines, with five syllables in the first line, seven in the second, and five in the third. It does not need to rhyme. Only return the haiku, no image description is necessary. Please insert a * between each line instead of a comma (,)."
CABLE_PROMPT="This image features a type of cable connector. Please specifically identify the cable connection. You should describe the category of the cable and the specific type. For example, if you see a USB cable, you should also say whether it is type A, type mini B, etc. If you are not sure what type of cable connector it is, please state that and make your best inference. The response should be concise, no more than two sentences. You do not need to describe the general appearance of the image."
TRANSLATE_PROMPT="This image has text that is not in English. Please translate it to English. Only return the translated text in English. For example, if the text reads hola (hello in Spanish), then return hello (translated to English). No further explanation is necessary. If you are not sure on how to translate a word(s) then please return the text as seen in the image. You do not need to describe the image."
ALIEN_PROMPT="Describe this image as if you are a friendly alien visiting from another world in our galaxy. You have no context for society on Earth. Your descriptions are cute, inquisitive and concise (2-3 sentences). You do not need to comment on the quality of the image."
WEIRD_PROMPT="Take in the full image as context. Is there anything in the image that is out of the ordinary or interesting? For example, if the image is of a backyard, do you see any animals or objects that catch your eye? You do not need to comment on the quality of the image. Please keep responses brief and concise, between one to two sentences. If you do not see anything out of the ordinary for the context of the image or anything particularly interesting, please return: Nothing to see here."

About the Prompts

The text prompts were written to try and cover a wide range of possibilities. You'll see a recurring theme of asking OpenAI to be concise and to not comment on the image quality. These were consistent issues that popped up when constructing a prompt and should be considered if you write your own prompts. The main takeaway for writing prompts in general is that you should not assume any context on the part of OpenAI. If there is an important caveat or need for your prompt, then you should include it. The more detail, the better.

Alt Text Prompt

The alt text prompt is the most general prompt included. It returns a brief description of the image. Mastodon was included as a reference point because of that social media network's commitment to accessibility and the prolific use of alt text on the site.

Haiku Prompt

This is one of the sillier prompts included. It requests a haiku be written about the main subject of the image. The guidelines on what a haiku is is included because it was initially returning responses that had too many syllables to be a haiku. The * separator is requested so that it can be used as a line break in the CircuitPython code when the response is displayed on the MEMENTO screen.

Cable Identifier Prompt

This is an attempt at a machine learning-style response. It asks OpenAI to identify a cable connection, such as a USB-A cable or a TRS audio cable.

Translation Prompt

This prompt asks that any text in the image be translated to English. It was tested using common phrases in multiple languages via Google translate on a computer screen, language flashcards, product directions and hand written notes to check if it was a viable idea.

Alien Prompt

This prompt is purely for entertainment. It asks that the image be described from the point of view of an alien visiting from outer space. Guidelines were included to make sure that the responses were wholesome, such as noting that the alien is friendly and that the descriptions are cute and inquisitive.

Is Anything Out of Place? Prompt

Like the cable identifier prompt, this prompt attempts a pseudo-machine learning result by asking for OpenAI to determine if anything is interesting in the image based on the context of the image. An application for this could be a wildlife camera or monitoring for packages being delivered on a porch.

Page last edited