Iterators

circuitpython_yomex-owo-640069-unsplash.jpg
Photo by Yomex Owo on Unsplash

Comprehensions are not without their problems. They require all of the items in the list to be in memory and they create another list in memory for the result. This works fine for smallish lists (depending on how much memory you have available) but don't work as well with large amounts of data and/or memory constrained environments.  This is where iterators can be useful.

One big advantage of an iterator is that the items aren't all in memory; only one is at a time. As stated above, that's not much of an advantage with small lists.  But it should be clear that this is a huge advantage for large lists; it lets you minimize the amount of memory required. Furthermore, since they only use one item at a time, they're great when you don't know how many items there are. One example of this is if you are processing logged data from a file, or an ongoing series of readings from a sensor. Iterators work by only computing the next value when it's asked for, and only the next one. Another advantage is that iterators allow us to pass them around in our code, pulling out values as needed. Finally, because each value is computed only when it's asked for, an iterator can provide an infinite sequence of values.

Python's for loop uses iterators, or rather iterables, behind the scenes. The general form of the for loop is:

for variable in iterable:
    ...

So lists, ranges, and strings are all iterables. However, to iterate over them you need to use a for loop. An iterator provides a more general way of getting a value from an iterable, then getting the next value, then the next, and so on. The iter function wraps an iterator around an interable, allowing us to step through its values as we choose without having to use a for loop. We pass the iterator to the next function to, as you might expect, get the next value.

If we try to use next() with ranges and lists, it clearly doesn't work.

Download: file
r = range(10)
next(r)
#Traceback (most recent call last):
#File "", line 1, in 
#TypeError: 'range' object is not an iterator
l = [1, 2, 3]
next(l)
#Traceback (most recent call last):
#File "", line 1, in 
#TypeError: 'list' object is not an iterator

Wrapping in iterator around them will let this work.

Download: file
r = iter(range(10))
next(r)
#0
next(r)
#1
l = iter([1, 2, 3])
next(l)
#1
next(l)
#2
This guide was first published on Oct 11, 2018. It was last updated on Oct 11, 2018.
This page (Iterators) was last updated on Jul 15, 2020.