What is a Jupyter notebook? It is a collection of "notes" (cells) in Markdown format, code, results, and media, commonly used to tweak and display information and for presenting complex data analysis.

It's also the way you interact with a programming backend system (kernel) to run the code from those notebook cells, but setting up a kernel can sometimes turn out to be a pain, so you might be relieved to know you don't have to!

These days it's all about online tools, install nothing manually, and here are a few suggestions for you. But there are also local editor options included, so everyone should find something suitable for their needs, otherwise leave guide feedback, or post questions on the forums / Discord.

As a first play, it's worth visiting this page of officially recommended online notebook platforms:

https://jupyter.org/try

Also note that although it's traditional to use a Python kernel with Jupyter Notebooks (originally called IPython notebooks), many other programming languages are supported via additional kernels.
See the Jupyter page for more info:  https://jupyter.org/try#kernels

Python first, but let's talk about another kind of snake... Anaconda

If you're new to all this, the Jupyter Project recommends installing Anaconda. Anaconda is a collection of data science tools, installable for all platforms, that includes everything required for Jupyter Notebooks (like Python and Jupyter, and the IPython kernel). It also comes with more sophisticated versions like Jupyter Lab, and support for running notebooks in VS Code and other editors.

They also have a cloud platform, if you wish to try that first, at anaconda.cloud, which has a free plan that comes with a small but useful AI chatbot allowance.

Navigate to the Anaconda downloads page, select to skip registration, then select your operating system, and download the installer including Python 3.12+.

Install the version of Anaconda you downloaded by following the provided instructions.

Launching Jupyter Notebook

Once Anaconda is installed, open the Anaconda Navigator Application.

You will be presented with a panel of icons to launch different software options. You will see the option to launch Jupyter Notebooks from the bundled launcher. 

Launching Jupyter notebooks will give you a web page to interact with the notebooks, but notice that there are a lot of other options included in the launcher (some of which come with AI assistance).

Using a Notebook and adding the data

When you first load a notebook, the cells may already have notes (text and images) and code cells, possibly with the results of the code cells presented beneath them. That's because notebooks allow saving results too!

Assuming you have some code that has not been run since being updated (or had the results cleared / not saved), then after selecting the cell row (left hand blue bar indicates the current cell), click the play icon or choose Run Cell(s) from the menu toolbar which will ask the kernel to run the code and present the results.

Notice the Fast Forward icon which means restart kernel and run all cells. This sometimes has problems so doing the individual cells with the play button is most reliable, try combining with Select all cells.

Great, now that you understand how to execute the code cells, we can begin our data exploration journey.

Download the provided notebook above, unzip its contents, and open in your Jupyter Notebook platform, Usually this requires uploading the notebook file first if using an online platform, otherwise just open the notebook file (with the .ipynb file extension).

First we need to add the data file, usually by uploading a CSV file or equivalent (like a JSONL formatted .log file), but you can also just include anything you like as code (so in this case we'll use dummy data instead of a file), or possibly fetch a file from a URL (if your notebook platform allows that and most do).

After uploading the data, you need to add some code in a cell to import the necessary Python libraries. It's also a good time to install any extra requirements, using the syntax !pip install X where X is the library name.

We're going to install some later, but for now we'll import a couple of libraries that come bundled with Jupyter Notebooks to show some of the basics.

The main two number and data manipulation libraries are Numpy and Pandas. It's common to give them a shorter alias instead of their full name, so the import statements in the code cell will look like this:

# Comments begin with a hash symbol in Python 

import numpy as np 
import pandas as pd

Next to get a library to provide a more visual output instead of just printed text, so a graphing or visualisation library is imported (there are many alternatives to matplotlib):

import matplotlib

Finally, to explore our data... You'll need to appreciate that pandas operates on what's called Data Frames (think of them as rows), and is a wizard at manipulating and understanding data. It works in partnership with numpy, which provides support for mathematical routines and complexity along with optimised algorithms and its own more efficient data types to represent the standard Python data types. This is why later you will see some of data becomes np.nan (NaN = Not a Number) instead of blank values or empty strings ("").

In the provided notebook, there is a section in the code cell at the top that includes some fake data (the lines after records = that look like JSON data). The code then checks if those records are commented out, and uses the log file instead (specified by the log_file_path variable), or whether to use the fake data. If you don't have any data file then uncomment the records lines to include the sample data.

Start by converting the data to a Pandas DataFrame. There are some handy methods to automatically import a CSV using pd.read_csv(filename), or JSON/JSONL file using pd.read_json(filename, lines=True), and it converts them automatically into data frame objects, so take advantage when you can.

It's not too hard to convert things manually, and that will be done for JSONL data to ensure that each line is valid and alert you to which lines are broken, if detected (possibly if the logger's batteries were depleted).

Now that the data is in DataFrames, you can start slicing it up or adding extra columns. The whole dataframe collection is treated as a single object at times, where using the indexing notation dataframe_object[index] will refer to whole columns.

It can also refer to a collection of testable items at other times, where the syntax is used to filter / test the objects before returning a subset.

Both will be later, but for now just the first syntax will be used to convert the timestamp column, and then to display the first ten rows:

Great, now let's plot some data, but what data to plot... (Surely it cannot be displayed all at once!)

It's worth looking briefly at the code cells above the timestamp conversion, as there the components (sensors) are given a Component Name column ("component_name") which allows you to easily disambiguate (separate) the different data streams by sensor.

If the config.json file is provided, then the components will receive full sensor names, otherwise the automatically generated names will be based on the Pin or I2C address information.

You can see in the image above that the component names are just using the automatic versions listing pin name etc, because no config.json file was provided for the initial run.

Since the cells were executed, a correct config file has been uploaded, but the cells/notebook will need to be executed again to update with the new values.

You can use the component_names to filter the data, even updating the graphs automatically!

Self-adjusting Charts - Can't we have interactive widgets?

You can! There is a fantastic documentation page for the IPyWidgets project (now Jupyter-Widgets), which demonstrates the widgets in interactive notebooks using the Pyodide kernel (running Python in the browser), and has many example notebooks to demonstrate different usage.

Use the play icon to work through the notebooks one cell at a time.

See https://ipywidgets.readthedocs.io/     (I was impressed to see a Game Controller input widget)

Here the dropdown widget is used to allow selecting the sensor and measurement type / unit, along with a slider to filter the date range if desired:

Later in the notebook, there are a couple of cells that show how to filter a dataset, and then add a few columns for data analysis reasons (to show rate of change).

Have a quick look, and maybe it will inspire you to take your data a little further...

All of these examples were tested using locally installed Jupyter Notebooks, along with on the various try-it-now Jupyter Labs online platforms (using Jupyter-lite), and additionally in the Anaconda Cloud platform.

AI-Assisted Data Analysis and Alternative Editors

Editor preferences vary, and personally I'm more of a Visual Studio Code (VSCode) fan, which does require adding the official Jupyter extension to VSCode before use.

Visual Studio Code is my preferred choice because it supports editing most things flawlessly, and now has a free Copilot chat option that can do remarkably well at following your careful instructions, generating the code for each notebook cell with relative ease. 

The key is to include the right (just enough) context by dropping in relevant files along with your written instructions into the Copilot chat panel. Similarly, provide it any error messages to have it attempt to fix them.

There are offline assistants, too, if you want the best of both worlds. They aren't as quick offline so it can be frustrating. I did take a quick look at lumen which offers an offline supporting chat-based data analysis and visualisation platform. It also supports any model on huggingface.co which is useful.

To better understand using notebooks in VS Code, I recommend reading the official guide (with helpful video):

https://code.visualstudio.com/docs/datascience/jupyter-notebooks

Other Online Platforms: Google Colab

This is perhaps the oldest platform for freely hosted Jupyter Notebooks, Colab (or 'Colaboratory') was first introduced in 2017 as a research project by Google. It comes with a generous allowance of compute per month, storage, memory, and even GPU capable machines for more powerful work. That means you can run some complex things that normally require expensive graphics cards, like machine learning / inference jobs, or photogrammetry (meshroom). They also host a large collection of public notebooks and datasets.

Check out this interactive notebook that explains getting started, and links to further info for charts / visualisation, importing data, and many many more things.

Another online platform - Deepnote.com

Worth a mention as I've been using Deepnote since 2021, and it has a great data visualisation and exploration tool built-in.

This allows one to explore the data in a dataframe without having to plot it first. Also facilitating pivot-tables (re-stratifying your data), and normal graphs. It additionally comes with an AI assistant that can help push you in a good starting direction with your data.

What about Live Data in notebooks?

Adafruit has a couple of great guides on using Jupyter Notebooks for data analysis, with live data flowing into the notebook from a sensor, and you might be able to use the same hardware you have now to do similar, if desired, (it's just Python code reading USB Serial data into the PC).

Check out the guide linked here which uses an MCP2221 to convert USB to General Purpose Input Output (GPIO) pins along with an I2C interface over Stemma QT, and the other related guide for running a CircuitPython kernel in a Jupyter Notebook (and using a CircuitPlayground Express to provide interactive data):

And that about wraps up this guide. Good luck on your future data exploration journeys!