Skip to main content

Chainable iterators (sync and async) for TypeScript, with support for opt-in, bounded parallelism

This package works with Node.js, Deno, BrowsersIt is unknown whether this package works with Bun, Cloudflare Workers
JSR Score
88%
Published
4 months ago (1.5.0)

Better Iterators

This module provides Lazy and LazyAsync classes which make it easy to chain together data transformations.

The lazy function is the simplest way to get started.

import { lazy, range } from "./mod.ts"

// You can use any Iterable here (such as an Array, or Generator), but
// Lazy objects are themselves Iterable:
let iterable = range({to: 1000})

let results = lazy(iterable)
    .filter(it => it % 2 == 0)
    .map(it => it*it)
    .limit(10)

// No iteration has happened yet.
// This will trigger it:
for (let item of results) { console.log(item) }

Lazy Iteration Consumes All Input

Note that iterating a Lazy(Async) will consume its items -- the operation is not repeatable. If you need to iterate multiple times, save your result to an array with Lazy#toArray

import { lazy, range } from "./mod.ts"

let results = lazy([1, 2, 3, 4, 5]).map(it => `number: ${it}`)

for (let result of results) { console.log("first pass:", result)}

// Has no more values to yield, will throw an exception:
for (let result of results) { console.log("second pass:", result)}

Other iterator libraries show examples of parallel/async iteration like this:

import { lazy, range } from "./mod.ts"
let urls = [
    "https://www.example.com/foo",
    "https://www.example.com/bar"
]
let lazySizes = lazy(urls)
    .map(async (url) => {
        let response = await fetch(url)
        return await response.text()
    })
    // The type is now Lazy<Promise<string>> so we're stuck having to deal
    // with  promises for the rest of the lazy chain:
    .map(async (bodyPromise) => (await bodyPromise).length)
    // and so on...

// `lazySizes` also ends up as a `Lazy<Promise<number>>`, so we've got to 
// await all of the items ourselves:
let sizes = await Promise.all(lazySizes.toArray())

This approach might seem to work, but it has unbounded parallelism. If you have N URLs, .toArray() will create N promises, and the JavaScript runtime will start making progress on all of them simultaneously.

That might work for small workloads, but network and memory resources are not unbounded, so you may end up with worse, or less reliable performance.

Lazy Asynchronous Iteration

Better Iterators provides a simpler, safer API when working with async code: You can convert a Lazy to a LazyAsync:

import { lazy } from "./mod.ts"
let urls = [
    "https://www.example.com/foo",
    "https://www.example.com/bar"
]
let lazySizes = lazy(urls)
    .toAsync()
    .map(async (url) => {
        let response = await fetch(url)
        return await response.text()
    })
    // here the type is LazyAsync<string> (not Lazy<Promise<string>>)
    // so further lazy functions are easier to work with:
    .map(it => it.length)

Here, LazyAsync#map does the least surprising thing and does not introduce parallelism implicitly. You've still got serial lazy iteration.

If you DO want parallelism, you can explicitly opt in like this:

import { lazy } from "./mod.ts"
let urls = [
    "https://www.example.com/foo",
    "https://www.example.com/bar"
]
let lazySizes = lazy(urls)
    .map({
        parallel: 5,
        async mapper(url) {
            let response = await fetch(url)
            return await response.text()
        }
    })
    // etc.

Functional Idioms

You'll find other common Functional Programming idioms on the Lazy and LazyAsync types, like associateBy, groupBy, fold, sum, partition, etc.

Add Package

deno add @nfnitloop/better-iterators

Import symbol

import * as mod from "@nfnitloop/better-iterators";

Add Package

npx jsr add @nfnitloop/better-iterators

Import symbol

import * as mod from "@nfnitloop/better-iterators";

Add Package

yarn dlx jsr add @nfnitloop/better-iterators

Import symbol

import * as mod from "@nfnitloop/better-iterators";

Add Package

pnpm dlx jsr add @nfnitloop/better-iterators

Import symbol

import * as mod from "@nfnitloop/better-iterators";

Add Package

bunx jsr add @nfnitloop/better-iterators

Import symbol

import * as mod from "@nfnitloop/better-iterators";