Sequences and iterators¶

Rolling averages¶

class satella.coding.sequences.RollingArithmeticAverage(n: int = 100)¶

A class to implement a rolling arithmetic average over n last entries

Parameters:	n – amount of last entries to count

avg() → float¶

Compute current average

Returns:	current average
Raises:	ZeroDivisionError – the average buffer is empty

clear() → None¶: Clear the rolling average buffer

insert(x: float) → None¶

Add a value to the rolling average, discarding the previous entry if the buffer size is exceeded

Parameters:	x – sample to insert

Standard routines¶

length¶

satella.coding.length(lenable) → int¶: Return length of an item. If it is a generator, exhaust it and return it’s length.

IteratorListAdapter¶

class satella.coding.sequences.IteratorListAdapter(iterator: Iterator[T_co])¶

A wrapper around an iterator that enables it to be processed as a list.

Ie. the generator will now support __contains__, __len__ and __getitem__. If a call to such a method is done, the generator will be unfolded in memory so this might take a ton of memory! You’ve been warned!

Deprecated since version 2.15.7: Use :class:`~satella.coding.sequences.ListWrapperIterator ` instead

Parameters:	iterator – iterator to unfold

to_iterator¶

satella.coding.sequences.to_iterator(fun)¶

Convert function to an iterator. You can replace the following code:

>>> def iterator(x):
>>>     for y in x:
>>>         yield fun(y)

with

>>> @to_iterator
>>> def fun(y):
>>>     ...

and now call fun instead of iterator. fun will accept a single argument - the iterable, and assume that the function you decorate also takes a single argument - the item

map_list¶

satella.coding.sequences.map_list(fun: Callable, iterable: Union[Iterator[T], Iterable[T]]) → List[T]¶

A syntactic sugar for

>>> list(map(fun, iterable))

Parameters:	fun – function to apply iterable – iterable to iterate over

unique¶

satella.coding.sequences.unique(lst: Union[Iterator[T], Iterable[T]]) → Iterator[T]¶

Return each element from lst, but return every element only once.

Take care for elements of T to be __eq__-able and hashable!

This will keep internally a set of elements encountered, and skip them if same element appears twice

Parameters:	lst – iterable to process
Returns:	a generator yielding unique items from lst

iterate_callable¶

satella.coding.sequences.iterate_callable(clbl: Callable[[int], V], start_from: int = 0, exc_classes=(<class 'IndexError'>, <class 'ValueError'>)) → Iterator[V]¶

Given a callable that accepts an integer and returns the n-th entry, iterate over it until it starts to throw some exception.

Parameters:	clbl – callable to call start_from – number to start from exc_classes – exceptions that being thrown show that the list was exhausted
Returns:	an iterator

choose¶

To return the first single element that returns true on given callable, use the following function:

satella.coding.sequences.choose(filter_fun: Callable[[T], bool], iterable: Union[Iterator[T], Iterable[T]], check_multiple: bool = False) → T¶

Return a single value that exists in given iterable.

Essentially the same as:

>>> next(iter(filter(filter_fun, iterable)))

but raises a different exception if nothing matches (and if there are multiple matches and check_multiple is True). If check_multiple is True this guarantees to exhaust the generator (if passed).

Parameters:	filter_fun – function that returns bool on the single value iterable – iterable to examine check_multiple – if True, this will check if there are multiple entries matching filter_fun, and will raise ValueError if so. If True, this will exhaust the iterator. If left at default, False, this may not exhaust the iterator.
Returns:	single element in the iterable that matches given input
Raises:	ValueError – on multiple elements matching (if check_multiple), or none at all

choose_one¶

Does the same thing as choose, but exhausts the generator and checks if there are no multiple elements matching the callable. If there are, raises ValueError.

satella.coding.sequences.choose_one(filter_fun: Callable[[T], bool], iterable: Union[Iterator[T], Iterable[T]]) → T¶

Syntactic sugar for

>>> choose(filter_fun, iterable, check_multiple=True)

This exhausts the iterable.

Parameters:	filter_fun – function that returns bool on the single value iterable – iterable to examine
Returns:	single element in the iterable that matches given input
Raises:	ValueError – on multiple elements matching, or none at all

AlreadySeen¶

class satella.coding.sequences.AlreadySeen¶

Class to filter out unique objects. Objects must be hashable, barring that they must be eq-able, however passing it an non-hashable object will result in O(n^2) complexity, as the class uses a list to keep track of the objects.

Usage:

>>> als = AlreadySeen()
>>> for elem in sequence:
>>>     if als.is_unique(elem):
>>>         ... process the element ...

is_unique(key: K) → bool¶

Has the element been spotted first time?

Add it to the set.

Parameters:	key – element to check
Returns:	whether the element was seen for the first time

filter_out_nones¶

satella.coding.sequences.filter_out_nones(y: Sequence[T]) → List[T]¶

Return all elements, as a list, that are not None

Parameters:	y – a sequence of items
Returns:	a list of all subelements, in order, that are not None

index_of¶

satella.coding.sequences.index_of(predicate: Callable[[T], bool], seq: Sequence[T]) → int¶

Return an index of first met element that calling predicate on it returns True

Parameters:	predicate – predicate to apply seq – sequence to examine
Returns:	index of the element
Raises:	ValueError – if no element found

index_of_max¶

satella.coding.sequences.index_of_max(seq: Sequence[T]) → int¶

Return the index of the maximum element

Parameters:	seq – sequence to examine
Returns:	index of the maximum element
Raises:	ValueError – sequence was empty

f_range¶

satella.coding.sequences.f_range(*args) → Iterator[float]¶

A range() that supports float.

Note that this behaves correctly when given a negative step.

Call either:

>>> f_range(stop)   # will start from 0 and step 1
>>> f_range(start, stop)    # will start from start and continue until the result is gte stop
>>> # will start from start and continue by step until the result is gte stop
>>> f_range(start, stop, step)

Raises:	TypeError – invalid number of arguments

filter_out_false¶

satella.coding.sequences.filter_out_false(y: Sequence[T]) → List[T]¶

Return all elements, as a list, that are True

Parameters:	y – a sequence of items
Returns:	a list of all subelements, in order, that are not None

try_close¶

satella.coding.sequences.try_close(iterator: Iterator[T_co]) → None¶

Try to invoke close() on an iterator. Do nothing if provided iterator doesn’t have a .close() method.

Parameters:	iterator – iterator to close

n_th¶

satella.coding.sequences.n_th(iterator: Union[Iterator[T], Iterable[T]], n: int = 0) → T¶

Obtain n-th element (counting from 0) of an iterable

Parameters:	iterator – iterable to process n – element to return. Note that we’re counting from 0
Raises:	IndexError – iterable was too short

append_sequence¶

satella.coding.sequences.append_sequence(seq: Iterator[tuple], *elems_to_append) → Iterator[tuple]¶

Return an iterator which append elem_to_append to every tuple in seq.

Example:

>>> a = [(1, ), (2, ), (3, )]
>>> assert list(append_sequence(a, 1, 2)) == [(1, 1, 2), (2, 1, 2), (3, 1, 2)]

If every element of seq is not a tuple, it will be cast to one.

Parameters:	seq – sequence to append elems_to_append – element(s) to append
Returns:	an iterator

take_n¶

For the rare moments, when you wish you could just do:

iterator: tp.Iterator[T] = iterator
n_elements: tp.List[T] = iterator[:n]

But it doesn’t let you do this, because iterator is not subscriptable. However, this function comes to the rescue:

satella.coding.sequences.take_n(iterator: Union[Iterator[T], Iterable[T]], n: int, skip: int = 0) → List[T]¶

Take (first) n elements of an iterator, or the entire iterator, whichever comes first

Parameters:	iterator – iterator to take from n – amount of elements to take skip – elements from the start to skip
Returns:	list of p_len n (or shorter)

infinite_iterator¶

satella.coding.sequences.infinite_iterator(returns: Optional[T] = None, return_factory: Optional[Callable[[], T]] = None) → Iterator[T]¶

Return an infinite number of objects.

Parameters:	returns – object to return. Note that this will be this very object, it will not be copied. return_factory – a callable that takes 0 args and returns an element to return.
Returns:	an infinite iterator of provided values

is_instance¶

A factory for filter functions that check if given object is an instance of something (or multiple classes, if passed a tuple of classes). Use like that

orders: tp.List[BaseOrder] = ...
read_orders = filter(is_instance(ReadOrder), orders)

satella.coding.sequences.is_instance(classes: Union[Tuple[type, ...], type]) → Callable[[object], bool]¶

is_last¶

satella.coding.sequences.is_last(lst: Union[Iterator[T], Iterable[T]]) → Iterator[Tuple[bool, T]]¶

Return every element of the list, alongside a flag telling is this the last element.

Use like:

>>> for is_last, element in is_last(my_list):
>>>     if is_last:
>>>         ...

Parameters:	lst – list to iterate thru
Returns:	a p_gen returning (bool, T)

Note that this returns a nice, O(1) iterator.

enumerate2¶

satella.coding.sequences.enumerate2(iterable: Union[Iterator[T], Iterable[T]], start: int = 0, step: int = 1) → Iterator[Tuple[int, T]]¶

Enumerate with a custom step

Parameters:	iterable – iterable to enumerate start – value to start at step – step to add during each iteration

smart_enumerate¶

satella.coding.sequences.smart_enumerate(iterator: Union[Iterator[T], Iterable[T]], start: int = 0, step: int = 1) → Iterator[Tuple]¶

An enumerate that talks pretty with lists of tuples. Consider

>>> a = [(1, 2), (3, 4), (5, 6)]
>>> for i, b in enumerate(a):
>>>     c, d = b
>>>     ...

This function allows you just to write: >>> for i, c, d in enumerate(a): >>> …

Note that elements in your iterable must be either a list of a tuple for that to work, or need to be able to be coerced to a tuple. Otherwise, TypeError will be thrown.

Parameters:	iterator – iterator to enumerate start – value to start counting at step – step to advance the enumeration with
Raises:	TypeError – could not coerce the elements in your iterable to a tuple

smart_zip¶

satella.coding.sequences.smart_zip(*iterators) → Iterator[Tuple[T, ...]]¶

Zip in such a way that resulted tuples are automatically expanded.

Ie:

>>> b = list(smart_zip([(1, 1), (1, 2)], [1, 2]))
>>> assert b == [(1, 1, 1), (1, 2, 2)]

Note that an element of the zipped iterator must be a tuple (ie. isinstance tuple) in order for it to be appended to resulting iterator element!

Deprecated since version Use: the for (a, b), c syntax instead.

Parameters:	iterators – list of iterators to zip together
Returns:	an iterator zipping the arguments in a smart way

add_next¶

Sometimes you need to iterate through list and take also the next element.

satella.coding.sequences.add_next(lst: Union[Iterator[T], Iterable[T]], wrap_over: bool = False, skip_last: bool = False) → Iterator[Tuple[T, Optional[T]]]¶

Yields a 2-tuple of given iterable, presenting the next element as second element of the tuple.

The last element will be the last element alongside with a None, if wrap_over is False, or the first element if wrap_over was True

Example:

>>> list(add_next([1, 2, 3, 4, 5])) == [(1, 2), (2, 3), (3, 4), (4, 5), (5, None)]
>>> list(add_next([1, 2, 3, 4, 5], True)) == [(1, 2), (2, 3), (3, 4), (4, 5), (5, 1)]

Parameters:	lst – iterable to iterate over wrap_over – whether to attach the first element to the pair of the last element instead of None skip_last – if this is True, then last element, alongside with a None, won’t be output

half_cartesian¶

Sometimes you need just a half of your Cartesian product, for example for operations that are commutative (eg. checking for collisions, if object A collides with B then B collides with A).

It helps you save time during computationally intensive operations.

This routine will return a iterator of tuple containing two elements from the same set (ie. it will do something like a cartesian power of two).

skip_first¶

satella.coding.sequences.skip_first(iterator: Union[Iterator[T], Iterable[T]], n: int) → Iterator[T]¶

Skip first n elements from given iterator.

Returned iterator may be empty, if source iterator is shorter or equal to n.

Deprecated since version 2.14.22: Use itertools.islice instead

zip_shifted¶

satella.coding.sequences.zip_shifted(*args) → Iterator[Tuple[T, ...]]¶

Construct an iterator, just like zip but first by cycling it’s elements by it’s shift factor. Elements will be shifted by a certain factor, this means that they will appear earlier.

Example:

>>> zip_shifted(([1, 2, 3, 4], 1), ([1, 2, 3, 4], 0)) == [(2, 1), (3, 2), (4, 3), (1, 4)]

This will work on arbitrary iterators and iterables.

Shift can be negative, in which case the last elements will appear sooner, eg.

>>> zip_shifted(([1, 2, 3, 4], -1), ([1, 2, 3, 4], 0)) == [(4, 1), (1, 2), (2, 3), (3, 4)]

Same memory considerations as shift() apply.

The resulting iterator will be as long as the shortest sequence.

Deprecated since version 2.14.22: Use zip(shift(…)) instead

Parameters:	args – a tuple with the iterator/iterable and amount of shift. If a non-tuple is given, it is assumed that the shift is zero.

This is deprecated. Use zip(shift(...)) instead.

stop_after¶

satella.coding.sequences.stop_after(iterator: Union[Iterator[T], Iterable[T]], n: int) → Iterator[T]¶

Stop this iterator after returning n elements, even if it’s longer than that.

The resulting iterator may be shorter than n, if the source element is so.

Deprecated since version 2.14.22: Use itertools.islice instead

Parameters:	iterator – iterator or iterable to examine n – elements to return

group_quantity¶

satella.coding.sequences.group_quantity(length: int, seq: Union[Iterator[T], Iterable[T]]) → Iterator[List[T]]¶

Slice an iterable into lists containing at most len entries.

Eg.

>>> assert list(group_quantity(3, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10])) == [[1, 2, 3], [4, 5, 6],
>>>                                                                     [7, 8, 9], [10]]

This correctly detects sequences, and uses an optimized variant via slicing if a sequence is passed.

You can safely pass ranges

Parameters:	length – p_len for the returning sequences seq – sequence to split

iter_dict_of_list¶

satella.coding.sequences.iter_dict_of_list(dct: Dict[T, List[U]]) → Generator[Tuple[T, U], None, None]¶

Presents a simple way to iterate over a dictionary whose values are lists.

This will return the dictionary key and each of the value contained in the list attached to the key.

shift¶

satella.coding.sequences.shift(iterable_: Union[Reversible[T], Iterator[T], Iterable[T]], shift_factor: int) → Iterator[T]¶

Return this sequence, but shifted by factor elements, so that elements will appear sooner by factor.

Eg:

>>> assert list(shift([1,2, 3], 1)) == [2, 3, 1]

However note that this will result in iterators which have negative shift to be readed entirely into memory (converted internally to lists). This can be avoided by passing in a Reversible iterable.

Parameters:	iterable – iterable to shift shift_factor – factor by which shift elements.
Returns:	shifted sequence

other_sequence_no_longer_than¶

satella.coding.sequences.other_sequence_no_longer_than(base_sequence: Union[Iterator[T], Iterable[T]], other_sequence: Union[Iterator[T], Iterable[T]]) → Iterator[T]¶

Return every item in other_sequence, but limit it’s p_len to that of base_sequence.

If other_sequence is shorter than base_sequence, the shorter one will be returned.

Parameters:	base_sequence – sequence whose p_len should be taken other_sequence – sequence to output values from

count¶

satella.coding.sequences.count(sq: Union[Iterator[T], Iterable[T]], start: Optional[int] = None, step: int = 1, start_at: Optional[int] = None) → Iterator[int]¶

Return a sequence of integers, for each entry in the sequence with provided step.

Essentially the same (if step were ignored) as:

>>> (i for i, x in enumerate(sq, start=start_at))

Deprecated since version 2.14.22: Use start instead

Parameters:	sq – sequence to enumerate start – alias for start_at. Prefer it in regards to start_at. Default is 0 step – number to add to internal counter after each element start_at – deprecated alias for start
Returns:	an iterator of subsequent integers

length¶

satella.coding.sequences.length(iterator: Union[Iterator[T], Iterable[T]]) → int¶: Return the length of an iterator, exhausting it by the way

Even and odd¶

satella.coding.sequences.even(sq: Union[Iterator[T], Iterable[T]]) → Iterator[T]¶: Return only elements with even indices in this iterable (first element will be returned, as indices are counted from 0)

satella.coding.sequences.odd(sq: Union[Iterator[T], Iterable[T]]) → Iterator[T]¶: Return only elements with odd indices in this iterable.

Multirun¶

class satella.coding.sequences.Multirun(sequence: Iterable[T_co], dont_return_list: bool = False)¶

A class to launch the same operation on the entire sequence.

Consider:

>>> class Counter:
>>>     def __init__(self, value=0):
>>>         self.count = value
>>>     def add(self, v):
>>>         self.count += 1
>>>     def __eq__(self, other):
>>>          return self.count == other.count
>>>     def __iadd__(self, other):
>>>         self.add(other)
>>> a = [Counter(), Counter()]

The following:

>>> for b in a:
>>>     b.add(2)

Can be replaced with

>>> Multirun(a).add(2)

And the following:

>>> for b in a:
>>>     b += 3

With this

>>> b = Mulirun(a)
>>> b += 3

Furthermore note that:

>>> Multirun(a).add(2) == [Counter(2), Counter(2)]

Parameters:	sequence – sequence to execute these operations for dont_return_list – the operation won’t return a list if this is True

Generators¶

class satella.coding.sequences.ListWrapperIterator(iterator: Union[Iterator[T], Iterable[T]])¶

A wrapped for an iterator, enabling using it as a normal list.

The first time this is evaluated, list is populated with elements.

The second time, items are taken from the list.

It never computes more than it needs to.

Essentially a class that lets you reuse one-shot iterators.

This is additionally a generic class.

advance_to_item(i: int) → None¶: Makes the list be at least i in size

exhaust() → None¶: Load all elements of this iterator into memory.

next() → T¶

Get the next item

Raises:	StopIteration – next element is not available due to iterator finishing

class satella.coding.sequences.ConstruableIterator(*args, **kwargs)¶

An iterator that you can attach arbitrary things at the end and consume them during iteration. Eg:

>>> a = ConstruableIterator([1, 2, 3])
>>> for b in a:
>>>   if b % 2 == 0:
>>>       a.add(6)

All arguments you provide to the constructor will be passed to underlying deque

add(t: T) → None¶

Schedule given value to be iterated over after current items

Parameters:	t – value to iterate over

add_immediate(t: T) → None¶

Schedule given value to be iterated over during the next __next__ call

Parameters:	t – value to iterate over

add_many(t: Iterable[T]) → None¶

Schedule given values to be iterated over after current items

Parameters:	t – iterable of values

add_many_immediate(t: Iterable[T]) → None¶

Schedule given values to be iterated over during the next __next__ call

Parameters:	t – values to iterate over

satella.coding.sequences.walk(obj: T, child_getter: Callable[[T], Optional[List[T]]] = <class 'list'>, deep_first: bool = True, leaves_only: bool = False) → Iterator[T]¶

Return every node of a nested structure.

Parameters:	obj – structure to traverse. This will not appear in generator child_getter – a callable to return a list of children of T. Should return an empty list or None of there are no more children. deep_first – if True, deep first will be returned, else it will be breadth first leaves_only – if True, only leaf nodes (having no children) will be returned

satella.coding.chain(*args) → Iterator[T_co]¶

Construct an iterator out of provided elements.

If an element is an iterator, or an iterable it will be yielded-from. If it’s not, it will just be yielded.

A cast to iter() is used to determine iteratorness

satella.coding.exhaust(iterator: Iterator[T_co]) → None¶

Iterate till the end of the iterator, discarding values as they go

Parameters:	iterator – iterator to exhaust

class satella.coding.hint_with_length(generator: Generator[T_co, T_contra, V_co], length: Optional[int], length_factory: Optional[Callable[[], int]] = None)¶

Accepting a generator, return it additionally providing a specified __length_hint__

You can provide generator-generating functions as well

Parameters:	generator – generator to decorate length – length hint to provide length_factory – a callable called with no arguments to get the length

You must provide either length or length_factory. Giving them both is wrong, and will result in ValueError

Sometimes, you need the entire body of the generator to run. It’d be a shame if someone decided to bail out on you in the middle of the for loop. That’s what this class is for:

class satella.coding.SelfClosingGenerator(generator: Union[Generator[T_co, T_contra, V_co], Callable[[Any], Generator[T_co, T_contra, V_co]]])¶

A wrapper to exhaust the generator in response to closing it.

This will allow generators to complete that don’t provide a .close() method.

This will additionally exhaust the generator upon deallocation of the generator.

You can feed it with either generators, or generator-functions, it will behave correctly each time.

You can also use it as a context manager, to decouple finalizing the generator from the GC collection

Using it on your generator objects will assure that they will run to completion.

Take care: this won’t work on PyPy due to it’s nondeterministic garbage collection!

Deleters¶

Objects that allow you to easily (and rather quickly) remove elements from a list or a dict while iterating over them with minimum memory overhead.

ListDeleter¶

class satella.coding.ListDeleter(list_to_process: MutableSequence[T], direction: int = 0)¶

Having problems deleting entries from your list while iterating on them? No problem. Just swap the following:

>>> entries_to_delete = []
>>> for entry in my_list:
>>>     if entry.should_delete():
>>>         entries_to_delete.append(entry)
>>> for entry in entries_to_delete:
>>>     my_list.remove(entry)

With the following:

>>> with ListDeleter(my_list) as ld:
>>>     for entry in ld:
>>>         if entry.should_delete():
>>>             ld.delete()

You can also use the alternative syntax of: >>> ld = ListDeleter(my_list) >>> while True: >>> try: >>> v = ld.next() >>> except StopIteration: >>> break >>> if condition(v): >>> ld.delete() >>> ld.remove_items()

Note that a single ListDeleter running from a single context must be iterated on by only a single Thread as it keeps the state of iterator in itself, to prevent allocating new objects and slowing things down too much.

Note that calling reversed() on this will reset the pointer to the end of the list or the beginning of the list, respectively.

This allocates only a single object per a call to delete().

Calling the list deleter during iteration will yield the element.

You can pass any type of object here, as long as it supports pop(position) and __getitem__

next() → T¶

Returns:	the next element
Raises:	StopIteration – no more entries

prev() → T¶

Move to previous element, as per ordering.

Returns:	the previous element
Raises:	StopIteration – list is already at the first element!

remove_items() → None¶: After all of the items have been marked for deletion, delete them

DictDeleter¶

class satella.coding.DictDeleter(dict_to_process: collections.abc.MutableMapping)¶

Having problems deleting entries from your dict while iterating on them? No problem. Just swap the following:

>>> keys_to_delete = []
>>> for key, value in my_dict.items():
>>>     if value.should_delete():
>>>         keys_to_delete.append(key)
>>> for key in keys_to_delete:
>>>     del my_dict[key]

With the following:

>>> with DictDeleter(my_list) as ld:
>>>     for key, value in ld.items():
>>>         if value.should_delete():
>>>             ld.delete()

Note that a single DictDeleter running from a single context must be iterated on by only a single Thread as it keeps the state of iterator in itself, to prevent allocating new objects and slowing things down too much.

This allocates only a single object per a call to delete().