This guide has already covered the basics of CircuitPython library usage. Copy a library to your CIRCUITPY drive, and it's available for use in your code.
However, you may have noticed there are two bundles available for download, or that on some boards, you don't have to copy a library over to be able to use it. What gives? This page covers the deeper details about library files, how to use them, why to use one over the other, and much more.
Library files are available in two different file formats: .mpy and .py. There are separate library bundles for each file type. Libraries are accessible to CircuitPython in three different ways: .mpy files, .py files, or frozen modules. The next few sections detail the differences between the file types, and the various ways to access libraries.
RAM vs Filesystem Space
This page refers regularly to library file memory (RAM) usage. "Memory" on this page means RAM (Random Access Memory), which is the read/write memory in the microcontroller. RAM and memory are used interchangeably throughout this page.
There is one section that references filesystem space. Filesystem space is not RAM; it is where files are stored.
.mpy Library Files
The most commonly used library file type is an .mpy file. You can download the .mpy version of a library directly from its GitHub repository, but the simplest, most convenient way to get them is by downloading the appropriate bundle from the Libraries page on circuitpython.org. The first bundle on the Libraries page is the .mpy bundle. This bundle provides every library in the Adafruit CircuitPython Bundle in the .mpy format.
The Community Bundle, which is made up of community sourced and maintained libraries, is also available in an .mpy format, further down the Libraries page.
What is an .mpy file?
An .mpy file is a file that has been generated by running the mpy-cross tool on a .py file, which compiles the .py file to bytecode. This process removes the doc strings and does some minor optimisations. Further, comments, whitespace, and type hints don't translate into bytecode, so they are also not present following compilation. The result is a file that is smaller and uses less RAM memory. mpy-cross makes it possible to import some libraries that couldn't otherwise be imported as .py files on smaller boards because they take up too much RAM to compile on the board.
Basically, you can provide CircuitPython with an .mpy file, it is loaded into RAM, but as it is already compiled, it does not need to be compiled on the fly. Between that and the minor optimisation that comes with the .mpy format, it uses less memory than a .py file. One caveat is that .mpy files are not human-readable. If you want to look at the code directly, you'll need to use a .py file.
To use an .mpy formatted library, you simply copy over the .mpy library file to to the /lib folder on your CIRCUITPY drive. For example, if you're using a demo that requires the NeoPixel library, you would copy the neopixel.mpy file to your /lib folder.
If you're interested in modifying the library, you'll find that you can't edit an .mpy file. For that, you need to switch to a .py file.
Creating an .mpy File
Creating your own .mpy files out of .py files is a quick and simple process using mpy-cross.
Download the latest mpy-cross for your operating system from here, ensuring you choose the version of mpy-cross that matches your version of CircuitPython. Builds are available for Windows, macOS, x64 Linux, and Raspberry Pi Linux. For example, if you're using CircuitPython 7.3.0 with MacOS Big Sur, you could download mpy-cross-macos-bigsur-7.3.0-arm64. For ease of use, rename the file you download to mpy-cross.
On macOS and Linux, after you download mpy-cross, you must make the the file executable by doing chmod +x name-of-the-mpy-cross-executable.
To make an .mpy file, you will first need a .py file to work with. Open a terminal program or commandline, cd into the directory containing mpy-cross, and run the following:
./mpy-cross path/to/your-library-file.py
This will create your-library-file.mpy in the same directory as the original .py file.
Then, copy your-library-file.mpy to your CIRCUITPY drive to use it. Easy as that!
.py Library Files.
Every CircuitPython library is available as a .py file. You can download the .py library files individually from their respective GitHub repositories. The quicker, more convenient way is to download the .py version of the Adafruit CircuitPython Bundle from the Libraries page. It is available as the "Python Source Bundle".
The Community Bundle, which is made up of community sourced and maintained libraries, is available in a .py format, further down the Libraries page. It is also titled "Python Source Bundle".
What is a .py file?
A .py file is a Python file that you can open and modify in your favorite Python editor. However, unless you're planning to modify a library and test your modifications, it is, in general, better to stick with .mpy files. Basically, .py files require more memory, and trying to use them on smaller microcontrollers like the SAMD21 (M0) can result in memory allocation failures when the library is imported in your code.
Frozen Libraries
In some cases, library modules are "frozen" into the CircuitPython build for a specific board. You won't find a "frozen" bundle anywhere - there is nowhere to download frozen libraries, because they are built into CircuitPython for specific boards (i.e. only builds for boards that require frozen libraries). A library being frozen into CircuitPython means you can access that library without copying any files to CIRCUITPY/lib.
What is a frozen library?
Some microcontroller boards, such as the Circuit Playground Express, simply do not have enough memory to run code from some library files, even if they are .mpy format. In these cases, the applicable libraries and possibly the dependencies are included in the CircuitPython build for that specific board. Frozen libraries are any libraries that are included in CircuitPython builds for various boards. There are two advantages to doing this: it ensures code runs if there are otherwise memory issues, but more practically speaking, it means you can access these libraries in your code without copying the files to the CIRCUITPY/lib directory.
An excellent example of a board requiring frozen modules is the Circuit Playground Express. The CPX is easiest to use with the Adafruit CircuitPlayground Library. Many of the guides in the Adafruit Learn System for the CPX use the CircuitPlayground library. The problem is, if you copy the CircuitPlayground library, and its dependencies (Adafruit CircuitPython HID, LIS3DH, Thermistor, and NeoPixel) to the CPX, your code will quickly fail to run due to insufficient available memory. The CircuitPlayground Library imports all of the dependencies, so when you import it in your code, you're importing the rest of the list as well. So, what can you do? Freeze all of these libraries into the CPX CircuitPython build!
If you're wondering whether your microcontroller board has any frozen libraries in its CircuitPython build, you can check out the Downloads page on circuitpython.org for your specific board. Search for your board, and click on it to open its specific download page. Under each version of CircuitPython currently available for your board, you'll find a list of "Built-in modules available:", and, in the event that your build contains frozen libraries, a list of "Included frozen modules:", as well. The frozen modules list includes all libraries frozen into CircuitPython for your board. The following shows the PyPortal on circuitpython.org, including its list of frozen modules, highlighted in magenta in the bottom right corner of the image below.
Project Bundle and Frozen Libraries
Project Bundle basics are covered on the CircuitPython Libraries page. However, it's important to understand how the Project Bundle works with frozen libraries. The Project Bundle isn't aware of the frozen libraries. Therefore, the Download Project Bundle button downloads a zip containing all of the library files necessary for a given example, whether or not the libraries are frozen into CircuitPython for a particular board build. This won't cause issues with running the libraries (continue reading this page for details), but it does mean that the libraries are taking up filesystem space. If this causes issues for you, you can delete any library files from the /lib directory that are frozen into the CircuitPython build for your board.
Here is an example using the Circuit Playground Express. Below are the /lib directory from the the Project bundle in the Piano in the Key of Lime guide, and the list of frozen modules in CircuitPython for CPX as available on the Downloads page.
As you can see, all of the "Included frozen modules:" are also in the /lib directory in downloaded from the Project Bundle. Therefore, if you begin to run out of filesystem space on your Circuit Playground Express, you can delete the files from the /lib directory that match the list on the Downloads page to make more available. This applies to a project bundle for any microcontroller that contains frozen libraries..
Library File Priority
CircuitPython looks for specific library file types in specific locations on a microcontroller, both in a particular order. You can use this to your advantage when trying to use updated or modified libraries.
What if you want to test modifications or updates you made to a library file that is on your board as an .mpy file, or a library that is frozen into CircuitPython? There are a couple of ways to do this. To test your updates, you can copy the updated .py library file to the CIRCUITPY drive in the proper location. Specific to only frozen libraries, you can create your own build of CircuitPython with the change frozen in.
What library files does CircuitPython look for?
Library development requires the ability to test library changes live on a microcontroller to ensure they work properly. This means making an update to a .py file, copying it to CIRCUITPY, and running an example that uses your update. However, once you get into the development cycle, you may end up with multiple copies of a library on your board in different formats. How can you be sure CircuitPython is running the updated file? Check the format.
CircuitPython will attempt to run a .py file first. This means if you have a .py library file and a .mpy file of the same library in the same directory, CircuitPython will run the .py.
For example, if you have neopixel.py and neopixel.mpy in your /lib folder on CIRCUITPY, the neopixel.py file will take precedence over the neopixel.mpy file. Therefore, when you import neopixel in your code.py file to test your updates, it will import your updated library. The same results would occur if you had the same .py and .mpy file in root on your CIRCUITPY drive.
This can be used to your advantage, but it can also lead to potential issues. The Common Issues and Solutions section below has more details.
The location of various library files on CIRCUITPY also plays a part in how CircuitPython decides which file to choose. Find out the specifics in the next section.
Where does CircuitPython look for library files?
Testing your library modifications on a microcontroller is crucial. Frozen libraries had the potential for making that more difficult. as you cannot remove them from the board. So, CircuitPython looks for library files in different locations in a specified order, in a way that allows you to "override" libraries in other locations. How do you know what that order is? That's where sys.path comes in.
In CircuitPython (and Python), sys.path is a variable in the sys module that returns a list of strings that specify the search path for modules or libraries. More simply put, it allows for telling CircuitPython exactly where to look, and in what order, for library files on the CIRCUITPY drive.
To find out the specifics, you can run the following two lines of code in code.py or from the REPL on the board you're currently working with. If there are libraries frozen into CircuitPython for the particular board, the results will look like this:
>>> import sys >>> print(sys.path) ['', '/', '.frozen', '/lib']
If there are not frozen libraries, the results will look like this:
>>> import sys >>> print(sys.path) ['', '/', '/lib']
What does this actually mean? You can break down the results as follows.
If there are frozen modules built into the build of CircuitPython for the board you're using, CircuitPython looks for library files in the following locations in this exact order:
- Library files in the current directory (CIRCUITPY/)
- Library files in root (CIRCUITPY/)
- Fozen modules (.frozen is a fake directory for this purpose, as the frozen modules are not in the filesystem)
- Library files in the library folder (CIRCUITPY/lib)
If there are no frozen modules built into the build of CircuitPython for the board you're using, CircuitPython looks for library files in the following locations in this exact order:
- Library files in the current directory (CIRCUITPY/)
- Library files in root (CIRCUITPY/)
- Library files in the library folder (CIRCUITPY/lib)
What does all of this mean practically speaking? Here are a couple of examples.
- You put the .mpy version of a library into the /lib folder, and you find an issue. You obtain the .py version of the library file, make your changes, and are ready to test. Instead of deleting the .mpy in the /lib folder, you can copy your modified .py file to root on CIRCUITPY/, and it will "override" the .mpy as CircuitPython will find your modified .py first, and run it.
- You are using a microcontroller board with a particular library frozen into CircuitPython, and you find an issue. You obtain the .py version of the library, make your changes, and are ready to test. You can place the modified .py file in root on CIRCUITPY/, and it will "override" the frozen module as CircuitPython will find your modified .py first, and run it.
You can also run mpy-cross on your modified .py file, and place the .mpy file in root on CIRCUITPY/, and you will "override" the frozen modules and everything in the lib/ folder. This is necessary if you find that using the .py file causes your code to fail due to memory issues.
Ostensibly all of this also means you can put libraries in root on your CIRCUITPY/ drive, and they will run as they do in the /lib folder. However, if you always copy library files to the /lib folder, it means this technique for testing is always available without needing to delete files before you test.
Building CircuitPython with an Updated Frozen Library
With some boards, for example the Circuit Playground Express, you'll find that overriding the frozen modules with a .py file will cause running your code to fail due to a memory allocation failure. In some cases, even if you run mpy-cross on the .py file, your code will still fail. This means you may have to create your own build of CircuitPython with your modified library frozen into it.
The more difficult but most realistic way to test a frozen library modification is to build CircuitPython with your modified library included. It's the most realistic way to test because it ensures that the changes still fit into the specific CircuitPython build. It also allows you to test a library when, even as an .mpy, the code fails to run due to memory constraints. However, for some folks this is an intimidating prospect when they wanted to test a simple change they made to a library.
This is why there are detailed instructions for creating your own build of CircuitPython. Follow the instructions as is, up to the Build CircuitPython section. Before you run the actual build, you'll want to create a new branch in your local copy of the CircuitPython repository, and then copy the modified library into the frozen/ directory.
Before you can do that, you'll want to follow the steps here to clone a copy of the library you're modifying to your computer. Make your modifications within the local library directory. When you're ready to test, you'll copy the entire library directory, e.g. the Adafruit_CircuitPython_CircuitPlayground/ directory, into the frozen/ directory in root of your local copy of the CircuitPython repository. This will replace the existing version with your updated version.
Once you've updated the library directory in frozen/ with your modified version of the library, you're ready to build. Continue following the instructions in the Building CircuitPython guide to finish building your version of CircuitPython. Remember to change the board you're building for, if it is different from the one in the example. Once built, place your board into bootloader mode, and copy the firmware.uf2 file to the *BOOT drive.
Now, you're ready to test your changes!
What's the Real Difference?
You now understand the different ways libraries are supplied to CircuitPython, and which options work best for different use cases. But, how do you know that these separate options actually make a difference in memory usage? There's a simple way to see the actual difference, by the numbers, for yourself.
Memory Usage between .mpy and .py
There are two parts of importing a Python file that use extra memory. The code compilation process, and the resulting bytecode, which needs to be stored in memory during and after the compilation. Remember, .mpy files are already compiled, and do not need to be compiled on import.
Essentially, the import of an .mpy loads compiled code, and the import of a .py must compile the code on import. In both cases, the compiled code lives in RAM, and the resulting compiled code is basically the same size. However, the compilation process takes extra RAM temporarily during compilation, and if there's not enough RAM available, it will fail. Compilation may also cause some fragmentation, but this is mostly avoided in CircuitPython.
Think of it this way. You can bring a ready-to-eat cake to your friend's house, or you can show up with the plan to bake the same cake at your friend's house. Both completed cakes will take up the same amount of space. However, waiting to bake the cake means you'll need more space and resources before the cake is ready to eat, to combine the ingredients, bake it, and frost it. If it turns out your friend doesn't have the space available, or is missing a crucial ingredient, you wouldn't be able to complete the cake. Basically, if you're unsure whether you'll have everything needed to bake a cake at your friend's house, it's better to show up with a completed cake to ensure there is cake to be had!
If you're unsure whether your microcontroller board has the memory resources needed to run a .py version of a library, the simplest solution is to choose a .mpy file instead.
Using gc.mem_free() to Check Your Bytes
CircuitPython has a built-in gc (garbage collector) module, which includes gc.mem_free(). The mem_free() functionality is specific to CircuitPython (and MicroPython), and is therefore not available in the CPython gc module.
When you run gc.mem_free() in the REPL, or print(gc.mem_free()) in code.py, it returns the number of bytes of available heap RAM. This is the amount of RAM left for importing further libraries, and executing code. Specifically for our purposes, though, it will tell us how much memory is used when importing the same library as a .py file, an .mpy file, and a frozen module.
For this example, you will use the NeoPixel library on the Circuit Playground Express. The NeoPixel library is frozen into the CircuitPython build for CPX, as well as being available in the two bundles as neopixel.mpy and neopixel.py. To prepare for this, download both the mpy bundle and the Python Source Bundle. To get started, you'll want to connect to the serial console and enter the REPL.
For each of these examples, you'll first run the following from the REPL:
>>> import gc >>> gc.mem_free()
This provides you with a baseline of available memory before importing any further libraries.
First up, you'll test out how much memory is used when importing the frozen version of the NeoPixel library. Make sure you have no libraries copied anywhere on your CIRCUITPY drive, including in the lib folder!
Once you've determined the baseline, as explained above, run the following from the REPL:
>>> import neopixel >>> gc.mem_free()
As you can see, the baseline available memory is 17840 bytes. Once you import the frozen version of NeoPixel, you have 16432 bytes available. Importing the frozen version requires 1408 bytes.
Now, copy the neopixel.mpy file to the root of your CIRCUITPY/ drive. Remember, CircuitPython looks in the root directory for libraries before checking the frozen modules. This will ensure that you're importing the version you intend to.
Follow the same steps as above to get a baseline and show the memory available following import.
The baseline is the same, however, the available memory following import is now 13984 bytes. Importing neopixel.mpy requires 3856 bytes.
Next, delete the neopixel.mpy file from the CIRCUITPY drive. Copy the neopixel.py file to the root of your CIRCUITPY/ drive.
Follow the same steps as above to get a baseline and show the memory available following import.
The baseline is the same, however, the available memory following import is now 11680 bytes. Importing neopixel.py requires 6160 bytes.
Importing each file type requires significantly different amounts of memory! The available memory following each file type import is what's left to import more libraries, and run your code. You can see how using different file types might impact the ability to use memory-constrained microcontroller boards, and run more complicated code.
Common Issues and Solutions
This page has explained the different ways you can obtain, access, and test modifications to CircuitPython libraries. Here are some issues you may run into in the course of things, and suggestions as to how to deal with them.
.mpy Failing When It Worked Previously
You are certain that you downloaded the .mpy version of the library, and that the code ran previously using the .mpy file, but now, it's failing due to memory usage. The most common reason for this that there is a leftover .py version of the same library in the same directory (e.g. the .py and .mpy are both in /lib), or the .py file is in an "override" directory (e.g. the .py is in / and the .mpy is in /lib). In both of these examples, CircuitPython is going to load the .py file.
If you run into this, check out the contents of your CIRCUITPY drive, and ensure that you don't have any leftover .py files in / or in /lib. More often than not, you'll find that removing the .py file resolves this situation.
Memory Allocation
There are answers in the FAQ in this guide as to what a memory error is, and how to avoid it if you're interested in more detail. What it boils down to is your microcontroller board running out of memory to import libraries or execute code.
The Circuit Playground Express is a memory-constrained microcontroller board. Substituting the CircuitPlayground library for the NeoPixel library in the byte-checking example results in similar behavior for the frozen module and the .mpy file. However, if you try to import the CircuitPlayground library as a .py package, it will fail to import with an error similar to the one below.
The CircuitPlayground library is an excellent example of this because the library is quite large, has a significant number of dependencies imported in it, and absolutely cannot be run on the CPX as a .py file. This is not the case for all libraries frozen into all builds of CircuitPython (as shown with NeoPixel), so you will not always get this result when attempting to test modifications or updates.
Memory Fragmentation
A memory allocation failure does not necessarily mean you are entirely out of memory. CircuitPython memory can become fragmented the more you do, and have smaller chunks of memory available across the entire heap that add up to the amount needed, but not a single chunk big enough to allocate for a particular task. Fragmentation can block allocation of any chunk of memory that needs to be contiguous, such as a large buffer. In the context of importing a library file, it specifically means that there is not a single large enough chunk of memory to allocate the memory requested.
In the case of importing a .mpy file, the compiled bytecode is imported across multiple chunks. In the case of importing a .py file, the memory needed for compilation may exceed the size of the chunks available, in which case, you would run out of memory before there is even any bytecode to import.
CircuitPython does do some optimisation to avoid memory fragmentation. Once a library is imported successfully, it is moved to the end of the available memory, which decreases fragmentation. Some fragmentation is unavoidable over time, but the import optimisation definitely helps with avoiding as much initially.
Import Order Can Matter
If you have code that is failing on import, you may find that changing the import order allows your code to run. Before you have imported anything, there are larger chunks of memory available. Therefore, importing the larger libraries first, while there is still larger chunks of memory free, may enable your code to successfully run, even when it initially failed. This is not a guaranteed result, but in some cases, it can help.
Library Structure Not Intact or in Improper Directory
Remember that library files are available in two structures - a single file, and a directory containing multiple files. These structures must be kept intact for CircuitPython to be able to use the library. In the case of a single file, copy that standalone file to your CIRCUITPY drive. In the case of a directory, copy the entire directory and its contents to your CIRCUITPY drive.
Further, library files must not be placed into another directory. For example, if you create a neopixel/ directory on CIRCUITPY, and place neopixel.mpy inside of it, your code will fail. CircuitPython does not check superfluous directories for library files or packages.
In any of the situations above, your code will fail when you attempt to use the library in your code. When CircuitPython can't find the module you use within the library file, it will throw an error. The error will often be something like the following. (This error is specifically from an attempt to create the pixels object in code.py with neopixel.mpy in a neopixel/ directory.)
Don't Name Your Test Code the Same as a Library
Throughout your CircuitPython experience, you will write all sorts of different demos. You will obviously need to name files something other than code.py as you work through multiple demos, so you know what the file is, and so CircuitPython knows which file you currently want to run. However, in doing this, there is one serious caveat that can cause you all sorts of issues and make troubleshooting difficult. You must avoid naming your files the same name as a library file.
Say you write a NeoPixel example. Then you're ready to move on to a different example. So, you rename your current code.py to neopixel.py and move on to the next demo. The next time you need to use the NeoPixel library in your code, you will find that your code will fail. This is because it is trying to import your previously renamed code.py file, which of course does not have the library contents.
The best way to avoid this is to stay creative with your filenames. Do something like always include code in the name (i.e. neopixel_code.py), or get even more descriptive with it, calling it something like neopixel_all_pixels_red.py. These are two simple suggestions. There are also plenty of other ways to be creative about file naming.
Text editor powered by tinymce.
