Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions docs/reference/collections.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,80 @@ return annotations include this empty-input case. `sum_by` requires summable
projection results, not summable source elements; for example,
`Block.of("one", "four").sum_by(len)` returns `7`.

## Async callbacks

Use `asyncseq.amap` when the mapper returns an awaitable. It accepts ordinary
iterables and async iterables, and returns a lazy async iterable:

```python
from expression import pipe
from expression.collections import asyncseq


async def enrich_inputs(value: int) -> int:
return value * 2


async def example() -> list[int]:
stream = pipe(
[1, 2, 3],
asyncseq.amap(enrich_inputs),
asyncseq.filter(lambda value: value > 2),
)
return [value async for value in stream] # [4, 6]
```

The fluent equivalent is `AsyncSeq.range(3).amap(enrich_inputs)`. Constructing
either transformation requires no `await`; iteration awaits each mapper call.
Ordinary `pipe` is sufficient because the stages return async iterables, rather
than coroutines that must be resolved before passing to the next stage.

`asyncseq.map` applies a synchronous mapper; `asyncseq.amap` awaits its mapper.
Both preserve one output per input: an awaited list remains one list element,
unlike `collect`, which flattens inner sequences. Mapping is sequential and
preserves source order, with no task scheduling or prefetching. Mapper and source
exceptions, including cancellation, propagate to the consumer. Sources supporting
both iteration protocols are consumed asynchronously.

The same convention applies to the other callback-taking operations. Each has a
module-level function for pipelines and an `AsyncSeq` method:

| Operation | Awaited callback result | Output |
| --- | --- | --- |
| `amap(mapper)` | Mapped value | One item per source item |
| `afilter(predicate)` | Boolean | Items satisfying the predicate |
| `achoose(chooser)` | `Option` | Values from `Some`, skipping `Nothing` |
| `acollect(mapping)` | Ordinary or async iterable | Items from each inner iterable |
| `ascan(scanner, state)` | Updated state | Initial state followed by each update |
| `afold(folder, state)` | Updated state | Coroutine returning the final state |
| `afold_back(folder, state)` | Updated state | Coroutine returning the reverse fold |

All accept ordinary or async sources and await callbacks sequentially. Stream
transformations are constructed synchronously and return lazy async iterables;
the fluent forms return `AsyncSeq`, allowing chaining. `acollect` awaits the
callback and fully consumes its inner iterable before requesting another source
item. An async generator function already returns an async iterable directly, so
use the existing `collect` for that kind of callback.

`ascan` yields the initial state even for empty input, without advancing the source.
Both folds return the initial state for empty input. `afold` and `ascan` call their
callback with `(state, item)`; `afold_back` calls it with `(item, state)` in reverse
source order. Reverse folding buffers the entire finite source and therefore uses
memory proportional to its length.

```python
async def add(state: int, value: int) -> int:
return state + value


async def total() -> int:
return await pipe([1, 2, 3], asyncseq.afold(add, 0)) # 6
```

When a fold's resolved result must feed another pipeline stage, use `apipe` to
await it between stages. Operations without callbacks, such as `take`, `skip`,
and `length`, do not need another variant.

## Additional modules

```{eval-rst}
Expand Down
176 changes: 175 additions & 1 deletion expression/collections/asyncseq.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import builtins
import itertools
from collections.abc import AsyncIterable, AsyncIterator, Callable
from collections.abc import AsyncIterable, AsyncIterator, Awaitable, Callable, Coroutine, Iterable
from typing import Any, cast, overload

from expression.core import Option, pipe
Expand All @@ -16,6 +16,42 @@ async def map[TResult](self, mapper: Callable[[TSource], TResult]) -> "AsyncSeq[
# underlying async iterable and wrap the result back into an AsyncSeq.
return AsyncSeq(pipe(self._ai, map(mapper)))

def amap[TResult](self, mapper: Callable[[TSource], Awaitable[TResult]]) -> "AsyncSeq[TResult]":
"""Lazily await the mapper for each item, sequentially and in source order.

Constructing the sequence requires no await. Exceptions and cancellation
propagate during iteration; returned collections remain individual items.
"""
return AsyncSeq(amap(mapper)(self))

def afilter(self, predicate: Callable[[TSource], Awaitable[bool]]) -> "AsyncSeq[TSource]":
"""Lazily keep items whose awaited predicate is true."""
return AsyncSeq(afilter(predicate)(self))

def achoose[TResult](self, chooser: Callable[[TSource], Awaitable[Option[TResult]]]) -> "AsyncSeq[TResult]":
"""Lazily await the chooser and yield the values of present options."""
return AsyncSeq(achoose(chooser)(self))

def acollect[TResult](
self, mapping: Callable[[TSource], Awaitable[Iterable[TResult] | AsyncIterable[TResult]]]
) -> "AsyncSeq[TResult]":
"""Lazily await each mapping and flatten its ordinary or async iterable."""
return AsyncSeq(acollect(mapping)(self))

async def afold[TState](self, folder: Callable[[TState, TSource], Awaitable[TState]], state: TState) -> TState:
"""Consume the sequence, awaiting each accumulator update in order."""
return await afold(folder, state)(self)

async def afold_back[TState](self, folder: Callable[[TSource, TState], Awaitable[TState]], state: TState) -> TState:
"""Buffer the sequence, then await accumulator updates in reverse order."""
return await afold_back(folder, state)(self)

def ascan[TState](
self, scanner: Callable[[TState, TSource], Awaitable[TState]], state: TState
) -> "AsyncSeq[TState]":
"""Lazily yield the initial state and each awaited accumulator update."""
return AsyncSeq(ascan(scanner, state)(self))

@classmethod
def empty(cls) -> "AsyncSeq[Any]":
return AsyncSeq(empty())
Expand Down Expand Up @@ -289,6 +325,25 @@ async def _map(source: AsyncIterable[TSource]) -> AsyncIterable[TResult]:
return _map


def amap[TSource, TResult](
mapper: Callable[[TSource], Awaitable[TResult]],
) -> Callable[[Iterable[TSource] | AsyncIterable[TSource]], AsyncIterable[TResult]]:
"""Map an ordinary or async iterable with an awaitable-returning callback.

The returned async iterable is lazy: each requested item awaits one mapper
call before advancing the source. Results preserve source order and are not
flattened. No tasks are scheduled, and exceptions and cancellation propagate.
Use ordinary ``pipe`` to compose this transformation; consume it with
``async for``. Sources implementing both protocols use async iteration.
"""

async def _amap(source: Iterable[TSource] | AsyncIterable[TSource]) -> AsyncIterator[TResult]:
async for value in _iterate(source):
yield await mapper(value)

return _amap


def choose[TSource, TResult](
chooser: Callable[[TSource], Option[TResult]],
) -> Callable[[AsyncIterable[TSource]], AsyncIterable[TResult]]:
Expand All @@ -314,3 +369,122 @@ async def _collect(source: AsyncIterable[TSource]) -> AsyncIterable[TResult]:
yield item

return _collect


async def _iterate[TSource](source: Iterable[TSource] | AsyncIterable[TSource]) -> AsyncIterator[TSource]:
if isinstance(source, AsyncIterable):
async for value in source:
yield value
else:
for value in source:
yield value


def afilter[TSource](
predicate: Callable[[TSource], Awaitable[bool]],
) -> Callable[[Iterable[TSource] | AsyncIterable[TSource]], AsyncIterable[TSource]]:
"""Lazily await a predicate for each item, yielding only matching items.

Accept ordinary or async iterables. Callbacks run sequentially in source
order; exceptions and cancellation propagate during iteration.
"""

async def _afilter(source: Iterable[TSource] | AsyncIterable[TSource]) -> AsyncIterator[TSource]:
async for value in _iterate(source):
if await predicate(value):
yield value

return _afilter


def achoose[TSource, TResult](
chooser: Callable[[TSource], Awaitable[Option[TResult]]],
) -> Callable[[Iterable[TSource] | AsyncIterable[TSource]], AsyncIterable[TResult]]:
"""Lazily await a chooser and yield present values, skipping Nothing.

Accept ordinary or async iterables. Callbacks run sequentially in source
order; exceptions and cancellation propagate during iteration.
"""

async def _achoose(source: Iterable[TSource] | AsyncIterable[TSource]) -> AsyncIterator[TResult]:
async for value in _iterate(source):
result = await chooser(value)
if not result.is_none():
yield result.value

return _achoose


def acollect[TSource, TResult](
mapping: Callable[[TSource], Awaitable[Iterable[TResult] | AsyncIterable[TResult]]],
) -> Callable[[Iterable[TSource] | AsyncIterable[TSource]], AsyncIterable[TResult]]:
"""Lazily await each mapping and flatten its ordinary or async iterable.

Accept ordinary or async sources. Consume each inner iterable before advancing
the source. Exceptions and cancellation propagate. For callbacks returning
async iterables directly, without an awaitable wrapper, use ``collect``.
"""

async def _acollect(source: Iterable[TSource] | AsyncIterable[TSource]) -> AsyncIterator[TResult]:
async for value in _iterate(source):
async for item in _iterate(await mapping(value)):
yield item

return _acollect


def afold[TSource, TState](
folder: Callable[[TState, TSource], Awaitable[TState]], state: TState
) -> Callable[[Iterable[TSource] | AsyncIterable[TSource]], Coroutine[Any, Any, TState]]:
"""Consume an ordinary or async iterable, awaiting each accumulator update.

Return the initial state for empty input. Run callbacks sequentially in source
order, propagating exceptions and cancellation. Await the resulting coroutine.
"""

async def _afold(source: Iterable[TSource] | AsyncIterable[TSource]) -> TState:
result = state
async for value in _iterate(source):
result = await folder(result, value)
return result

return _afold


def afold_back[TSource, TState](
folder: Callable[[TSource, TState], Awaitable[TState]], state: TState
) -> Callable[[Iterable[TSource] | AsyncIterable[TSource]], Coroutine[Any, Any, TState]]:
"""Buffer a finite ordinary or async iterable, then fold in reverse order.

Await callbacks sequentially with item before state. Empty input returns the
initial state. Exceptions and cancellation propagate. Memory use is linear in
source length; await the resulting coroutine.
"""

async def _afold_back(source: Iterable[TSource] | AsyncIterable[TSource]) -> TState:
items = [value async for value in _iterate(source)]
result = state
for value in reversed(items):
result = await folder(value, result)
return result

return _afold_back


def ascan[TSource, TState](
scanner: Callable[[TState, TSource], Awaitable[TState]], state: TState
) -> Callable[[Iterable[TSource] | AsyncIterable[TSource]], AsyncIterable[TState]]:
"""Lazily yield the initial state and each awaited accumulator update.

Accept ordinary or async iterables. Yield the initial state without touching
the source. Callbacks run sequentially; exceptions and cancellation propagate.
"""

async def _ascan(source: Iterable[TSource] | AsyncIterable[TSource]) -> AsyncIterator[TState]:
result = state
yield result
async for value in _iterate(source):
result = await scanner(result, value)
yield result

return _ascan
Loading
Loading