@earthstar/earthstar@11.0.0-beta.1
Earthstar
**This version of Earthstar does not yet use Willow. The next version will, and is being worked on! **
Earthstar is a small and resilient distributed storage protocol designed with a strong focus on simplicity and versatility, with the social realities of peer-to-peer computing kept in mind.
This is a reference implementation written in Typescript. You can use it to add Earthstar functionality to applications running on servers, browsers, the command line, or anywhere else JavaScript can be run.
Detailed API documentation for this module can be found here.
This document is concerned with the usage of this module's APIs. To learn more about what Earthstar is, please see these links:
To learn more about running Earthstar servers, see README_SERVERS
To learn more about this codebase, please see ARCHITECTURE.
To learn about contributing to this codebase, please see CONTRIBUTING.
Table of contents
Importing the module
It can be imported via URL into a browser:
<script type="module"> import * as Earthstar from "https://cdn.earthstar-project.org/js/earthstar.web.v10.0.1.js"; </script>
Or Deno:
import * as Earthstar from "https://deno.land/x/earthstar/mod.ts";
Earthstar's web syncing does not work with version of Deno between 1.28.0 - 1.29.3 (inclusive) due to a regression in these versions' WebSocket implementation. Use Deno 1.30.0. or later.
or installed with NPM:
{`npm install earthstar`}
import * as Earthstar from "earthstar"; // Node and browser APIs are namespaced in the NPM distribution: import { ReplicaDriverWeb } from "earthstar/browser"; import { ReplicaDriverSqlite } from "earthstar/node";
We recommend the browser and Deno versions. This module has been built with many standard web APIs that have need to be polyfilled to work in Node.
Instantiating a replica
Replica
is the central API of this module. It is used to write and read data
to a locally persisted copy of a share's data, and much more besides.
To instantiate a replica, you will need knowledge of a share's public address.
import { Replica, ReplicaDriverMemory } from "earthstar"; const replica = new Replica({ driver: ReplicaDriverMemory(YOUR_SHARE_ADDRESS), shareSecret: YOUR_SHARE_SECRET, });
The shareSecret
property is optional. If we omit it, the replica will be
read-only.
Generating share keypairs
You can create new shares whenever you want.
import { Crypto } from "earthstar"; const shareKeypair = await Crypto.generateShareKeypair("gardening");
The result of this operation will either be a ShareKeypair
object with
shareAddress
and secret
properties, or a ValidationError
.
Persisting data with drivers
Replica
must always be instantiated with a driver. These drivers tell the
Replica
how to store and retrieve data, with different drivers using different
storage mechanisms.
Here are the available drivers:
ReplicaDriverMemory
(works in all environments, but only stores data in memory)ReplicaDriverWeb
(works in the browser, stores data with IndexedDB)ReplicaDriverFs
(works on runtimes with filesystem access, stores data with Sqlite and the filesystem)
Drivers are made of two sub-drivers: one for documents, and one for attachments (arbitrary binary data).
There are some extra document drivers not used in the default drivers:
DocDriverLocalStorage
(works in runtimes supporting the WebStorage API)DocDriverSqliteFFI
(works in Deno, stores data with a FFI implementation of Sqlite, requires using the--unstable
flag, and is faster than the default driver inReplicaDriverFs
)
These document drivers can be used like this:
const driver: IReplicaDriver = { docDriver: new DocDriverSqliteFfi(SHARE_ADDR, FS_PATH), attachmentDriver: new AttachmentDriverFs(FS_ATTACHMENTS_PATH), }; const replica = new Replica({ driver });
Writing data
Writing data requires two things:
- A replica configured with a valid share secret
- An author keypair
Author keypairs can be generated like this:
import { Crypto } from "earthstar"; const authorKeypair = await Crypto.generateAuthorKeypair("suzy");
The result will be a new AuthorKeypair
object with address
and secret
properties, or a ValidationError
.
With a valid author keypair you can write data using Replica.set
:
const setResult = await replica.set(authorKeypair, { path: "/my-note", text: "Saw seven magpies today", });
The result of this operation is either an IngestEvent
describing the
operation's success (or failure, if one of the parameters was invalid in some
way).
Wiping data
Once written, data can be removed by overwriting it:
await replica.set(authorKeypair, { path: "/my-note", text: "", });
Or with the convenience method:
await replica.wipeDocAtPath(authorKeypair, "/my-note");
Creating ephemeral documents
There is another way to remove written data without leaving any trace of it.
Ephemeral documents are held by replicas until a specified time, until at which point they are deleted.
await replica.set(authorKeypair, { path: "/my-temporary-note!", text: "I accidentally stepped on the strawberries.", deleteAfter: TIME_IN_MICROSECONDS, });
To set an ephemeral document, the path must contain a !
, and the
deleteAfter
property must be set with a timestamp in microseconds.
Querying data
There are many ways to get data back out of a Replica. The simplest one is
Replica.getAllDocs
:
const everything = await replica.getAllDocs();
The most powerful is Replica.queryDocs
:
const mostRecentlyEditedWikiPageDocs = await replica.queryDocs({ historyMode: "latest", filter: { pathStartsWith: "/wiki", }, limit: 10, });
Here are all the querying methods on Replica
:
getAllDocs
getLatestDocs
getAllDocsAtPath
getLatestDocAtPath
queryDocs
queryPaths
queryAuthors
Detailed API documentation for all of them can be found here.
Using document contents
The documents returned by queries are plain objects with the following shape:
type Doc = { /** Which document format the doc adheres to, e.g. `es.5`. */ format: "es.5"; author: AuthorAddress; text: string; textHash: string; /** When the document should be deleted, as a UNIX timestamp in microseconds. */ deleteAfter?: number; path: Path; /** Used to verify the authorship of the document. */ signature: Signature; /** Used to verify the author knows the share's secret */ shareSignature: Signature; /** When the document was written, as a UNIX timestamp in microseconds (millionths of a second, e.g. `Date.now() * 1000`).*/ timestamp: Timestamp; /** The share this document is from. */ share: ShareAddress; /** The size of the associated attachment in bytes, if any. */ attachmentSize?: number; /** The sha256 hash of the associated attachment, if any. */ attachmentHash?: string; };
Though most applications will probably only use the author
, text
, and
timestamp
properties.
Syncing with other peers
Syncing data with other peers requires adding your replica(s) to an instance of
Peer
:
import { Peer } from "earthstar"; const peer = new Peer(); // Pretend myReplica is an instance of `Replica` peer.addReplica(myReplica); peer.sync("https://my.server");
Peer.sync
can be passed another instance of Peer
or a valid URL of an
Earthstar server to sync with.
The two peers will only sync the replicas with shares they have in common.
The result of Peer.sync
can be assigned and used to monitor the progress of
the sync operation:
const syncer = peer.sync("https://my.server"); syncer.onStatusChange((newStatus) => { console.log(newStatus); }); syncer.isDone().then(() => { console.log("Sync complete"); }).catch((err) => { console.error("Sync failed", err); });
Using document attachments
Documents can be written along with some arbitrary data which is persisted as an
'attachment'. Whereas a document's text
field can hold a UTF-8 string of 8kb,
attachments can be of any kind of data and of any size.
// Here we use Deno.readFile to get a file's contents as a Uint8Array const imageData = await Deno.readFile("/Desktop/leaf.jpg"); await replica.set(authorKeypair, { path: "/images/pear-leaf.jpg", text: "A close-up of a leaf of a pear tree", attachment: imageData, });
The path must have a file extension e.g. .jpg
, .mp3
if it also has an
attachment.
If we were attaching a large amount of data, we would use a ReadableStream
instead:
// Here we use Deno.readFile to get a file's contents as a ReadableStream<Uint8Array> const videoFile = await Deno.open("/Desktop/little-mole.mp4"); await replica.set(authorKeypair, { path: "/videos/little-mole.mp4", text: "A close-up of a leaf of a pear tree", attachment: videoFile.readable, });
Retrieving attachments
If you already have a document with an attachment, you can use
Replica.getAttachment
:
const attachment = await replica.getAttachment(docWithAttachment);
The result of this operation will be a DocAttachment
with getBytes
and
getStream
methods, undefined (if our replica has not received a copy of this
attachment from other peers), or a ValidationError
in case getAttachment
was
passed a document which can't have an attachment.
It's also possible to add attachments to many documents at once:
const allDocs = await replica.getAllDocs(); const allDocsWithAttachments = await replica.addAttachments(allDocs);
allDocsWithAttachments
will be an array of all documents with an added
attachment
property. The type of this property will either be DocAttachment
,
undefined
, or ValidationError
.
Discovering other peers on the local network
Earthstar is able to automatically discover other Earthstar peers on the same network, and in turn advertise itself to others.
const discoveryLan = new DiscoveryLAN({ name: "My Computer" }); for await (const discoveryEvent of peer.discover(discoveryLan)) { if (discoveryEvent.kind === "PEER DISCOVERED") { discoveryEvent.sync(); } }
On Node, the
DiscoveryLAN
API is exported from "earthstar/node".On Deno, this feature uses unstable APIs and is not included with the default module exports. It can be imported from
https://deno.land/x/earthstar/src/discovery/discovery_lan.ts
. Usage also requires the--unstable
flag.
Subscribing to replica changes
There are many ways to subscribe to the many events a replica generates during its lifetime.
If you want to subscribe to updates in order to update a UI, the most ergonomic
API is ReplicaCache
:
import { ReplicaCache } from "earthstar"; const replicaCache = new ReplicaCache(myReplica); const allDocs = replicaCache.getAllDocs();
The caveat is that the first time a query method is called, it always returns
an empty result. To get new updates, you must subscribe to changes. In the
following example, we build a UI with a fictitious renderDocListUI
function
with the results of ReplicaCache.getAllDocs
:
function triggerUIRender() { const allDocs = replicaCache.getAllDocs(); renderDocListUI(allDocs); } replicaCache.onCacheUpdated(() => { triggerUIRender(); }); triggerUIRender();
The important thing to remember is that the callback to onCacheUpdated
will
never trigger until the cache has been queried at least once.
If you're not tracking changes for a UI, Replica.getQueryStream
returns a
ReadableStream
of QuerySourceEvent
, for documents matching a specific query.
This API is great for creating indexes.
// Create a query stream for all docs with paths starting with /chat // And include all existing documents and all newly created or synced documents const chatMessagesStream = replica.getQueryStream({ filter: { pathStartsWith: "/chat" }, }, "everything"); chatMessagesStream.pipeTo( new WritableStream({ write(event) { if (event.kind === "success" || event.kind === "existing") { console.log(event.doc.text); } }, }), );
Finally, Replica.getEventStream
returns a ReadableStream
of ReplicaEvent
,
which includes events for new document ingestions, document expirations,
attachment ingestions, attachment prunes, and events for when the replica is
about to close (and has closed).
Using common settings between clients
There are a number of configurations which most Earthstar applications will want to persist between runs:
- The shares used
- An author to using for signing documents
- Servers to sync with
Earthstar offers a SharedSettings
API which persists settings for these
between sessions in all runtimes supporting the WebStorage APIs:
const settings = new SharedSettings(); settings.author = authorKeypair; settings.addShare(myShareAddress); console.log(settings.servers);
It also offers a method which instantiates a new Peer
with replicas for all
shares already added:
const settings = new SharedSettings(); // Create a peer with all saved shares and sync once with all saved servers. const peer = settings.getPeer({ sync: "once", onCreateReplica: (addr, secret) => { return new Replica({ driver: new ReplicaDriverMemory(addr), shareSecret: secret, }); }, });
Checking for errors
Many functions in Earthstar return errors like ValidationError
. These errors
are not thrown, so it's good to check for them:
import { isErr } from "earthstar"; const result = replica.set(authorKeypair, { path: "/hey", text: "Hello", }); if (isErr(result)) { console.error( "Something went wrong when you tried to write some data!", result, ); }
Changing the cryptographic driver
The Deno and browser versions of this module are configured by default to use the fastest cryptographic libraries available to them. The Node version is not, however.
import { setGlobalCryptoDriver } from "earthstar"; import { CryptoDriverChloride } from "earthstar/node"; setGlobalCryptoDriver(CryptoDriverChloride);
Add Package
deno add jsr:@earthstar/earthstar
Import symbol
import * as mod from "@earthstar/earthstar";
---- OR ----
Import directly with a jsr specifier
import * as mod from "jsr:@earthstar/earthstar";
Add Package
npx jsr add @earthstar/earthstar
Import symbol
import * as mod from "@earthstar/earthstar";
Add Package
yarn dlx jsr add @earthstar/earthstar
Import symbol
import * as mod from "@earthstar/earthstar";
Add Package
pnpm dlx jsr add @earthstar/earthstar
Import symbol
import * as mod from "@earthstar/earthstar";
Add Package
bunx jsr add @earthstar/earthstar
Import symbol
import * as mod from "@earthstar/earthstar";