Skip to content

Commit

Permalink
README: clarify
Browse files Browse the repository at this point in the history
Co-authored-by: laury-lopes <laurylopes@gmail.com>
  • Loading branch information
ebonnal and laurylopes committed Dec 16, 2023
1 parent e790a15 commit e7b3d19
Showing 1 changed file with 36 additions and 25 deletions.
61 changes: 36 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,30 +22,26 @@ from kioss import Pipe
integers: Pipe[int] = Pipe(source=lambda: range(10))
```

A `Pipe` is an ***immutable*** `Iterable` that you instantiate with a function returning an `Iterable` (the data source).
Instantiate a `Pipe` by providing a function returning an `Iterable` (the data source).

## 4. Declare operations

### 4.a. Transformations
A `Pipe` is ***immutable***, meaning that applying an operation returns a new child pipe while the parent pipe remains unchanged.

There are 2 kinds of operations:
- **transformations**: to act on the pipe's elements
- **controls**: to configure the behaviors of the iteration over the pipe

As we said a `Pipe` is ***immutable*** meaning that applying an operation on it only returns a new child pipe, the parent pipe remaining unmodified.

```python
odd_squares: Pipe[int] = (
integers
.map(lambda x: x**2)
.filter(lambda x: x % 2 == 1)
.map(lambda x: x ** 2) # transformation
.filter(lambda x: x % 2 == 1) # transformation
.slow(freq=10) # control
)
```


### 4.b. Controls

Some operations do not act on the data itself but control the behavior of a future iteration over the pipe, here we will rate limit the iteration to a maximum of 10 odd squares per second.

```python
rate_limited_odd_squares: Pipe[int] = odd_squares.slow(freq=10)
```
All operations are described in the ***Operations guide*** section.

## 5. Iterate

Expand All @@ -61,23 +57,26 @@ for i in rate_limited_odd_squares:
...
```

But alternatively, a pipe also exposes a convenient method `.run` to launch an iteration over itself until exhaustion. It catches exceptions occurring during iteration and optionnaly collects output elements into a list to return. At the end it raises if exceptions occurred.
Alternatively, a pipe also exposes a convenient method `.run` to launch an iteration over itself until exhaustion. It catches exceptions occurring during iteration and optionnaly collects output elements into a list to return. At the end it raises if exceptions occurred.

```python
odd_squares: List[int] = rate_limited_odd_squares.run(collect_limit=1024)

assert odd_squares == [1, 9, 25, 49, 81]
```



---
# Operations guide

# ***Operations guide***

Let's keep the same example:
```python
integers = Pipe(lambda: range(10))
```

## Transformations
# A. Transformations
![](./img/transform.gif)

## `.map`
Expand All @@ -89,7 +88,7 @@ integer_strings: Pipe[str] = integers.map(str)
You can pass an optional `n_threads` argument to `.map` for a concurrent application of the function using multiple threads.

## `.do`
Defines the application of a function on parent elements like `.map`, but will yield the parent element unchanged instead of the result of the function: it applies a *side effect*.
Defines the application of a function on parent elements like `.map`, but the parent elements will be forwarded instead of the result of the function.

```python
printed_integers: Pipe[int] = integers.do(print)
Expand All @@ -106,17 +105,22 @@ pair_integers: Pipe[int] = integers.filter(lambda x: x % 2 == 0)

## `.batch`

Defines the grouping of parent elements into batches each yielded as a single output element.
Defines the grouping of parent elements into batches.

```python
integer_batches: Pipe[List[int]] = integers.batch(size=100, period=60)
```

Here a batch will be a list of 100 elements, or less if the pipe is exhausted or an exception occurred or more than 60 seconds has elapsed since the last batch has been yielded.
In this example a batch will be a list of 100 elements.

It may contain less elements in the following cases:
- the pipe is exhausted
- an exception occurred
- more than 60 seconds (`period` argument) has elapsed since the last batch has been yielded.

## `.flatten`

Defines the ungrouping of parent elements, assuming that the parent elements are `Iterable`s.
Defines the ungrouping of parent elements assuming that the parent elements are `Iterable`s.

```python
integers: Pipe[int] = integer_batches.flatten()
Expand All @@ -126,7 +130,7 @@ It also accepts a `n_threads` parameter to flatten concurrently several parent i

## `.chain`

Defines the concatenation of the parent pipe with other pipes. The resulting pipe yields the elements of one pipe until it is exhausted and then moves to the next one. It starts with the pipe on which `.chain`is called.
Defines the concatenation of the parent pipe with other pipes. The resulting pipe yields the elements of one pipe until it is exhausted and then moves to the next one. It starts with the pipe on which `.chain` is called.

```python
one_to_ten_integers: Pipe[int] = Pipe(lambda: range(1, 11))
Expand All @@ -139,7 +143,7 @@ one_to_thirty_integers: Pipe[int] = one_to_ten_integers.chain(
)
```

## Controls
# B. Controls
![](./img/control.gif)

## `.slow`
Expand All @@ -150,11 +154,11 @@ Defines a maximum rate at which parent elements will be yielded.
slowed_integers: Pipe[int] = integers.slow(freq=2)
```

The rate is expression in elements per second, here a maximum of 2 elements per second will be yielded when iterating on the pipe.
The rate is expressed in elements per second, here a maximum of 2 elements per second will be yielded when iterating on the pipe.

## `.observe`

Defines that the parent elements yielding process will be logged.
Defines that the iteration process will be logged.

```python
observed_slowed_integers: Pipe[int] = slowed_integers.observe(what="integers from 0 to 9")
Expand Down Expand Up @@ -184,3 +188,10 @@ safe_inverse_floats: Pipe[float] = inverse_floats.catch(ZeroDivisionError)
```

You can additionnally provide a `when` argument: a function that takes the parent element as input and decides whether or not to catch the exception.


---
---


*If you want more inspiration on how to leverage kioss, feel free to check the `./examples` folder.*

0 comments on commit e7b3d19

Please sign in to comment.