Storage backends

A backend is one string. Change it and the same code writes to a local directory, a MinIO container, or a production bucket. There is no server component to run either way.

Backend string	What it is	Python	JavaScript
`file:///abs/path`	A directory on disk	Read + write	—
`memory://`	In-process, discarded on exit. For tests	Read + write	—
`http://host/bucket`	Any S3-compatible endpoint	Read + write	Read, and write with a `put` backend
`https://host/bucket`	The same, over TLS	Read + write	Read, and write with a `put` backend

JavaScript always goes over HTTP. It has no filesystem path, so a dataset written to file:// has to be served before a browser or Node process can read it — and a plain static server can serve reads but not accept writes.

Local disk

The fastest way to work. Nothing to install, nothing to start:

python

ds = db.Dataset.create("photos", schema, backend="file:///tmp/dreamdb-quickstart")

To read that dataset from JavaScript, serve the directory:

bash

npx http-server /tmp/dreamdb-quickstart -p 8791 --cors

--cors matters only for browsers; a Node script can read a server without it.

MinIO

An S3-compatible server in a container. Use it when you want to exercise the real HTTP path locally, or to share a dataset across machines on your network:

bash

docker run -d --name dreamdb-minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=dreamdb \
  -e MINIO_ROOT_PASSWORD=dreamdbsecret \
  minio/minio:latest server /data --console-address ":9001"

docker exec dreamdb-minio mc alias set local http://localhost:9000 dreamdb dreamdbsecret
docker exec dreamdb-minio mc mb local/demo
docker exec dreamdb-minio mc anonymous set public local/demo

The backend string is then http://localhost:9000/demo. Making the bucket public keeps development simple; it also means anyone who can reach the port can read it.

S3, R2, B2, Wasabi

Any store that speaks S3-style GET and PUT works. Point at the bucket:

python

ds = db.Dataset.create("photos", schema,
                       backend="https://my-bucket.s3.us-east-1.amazonaws.com")

Writes are signed with SigV4, picked up from the standard environment variables:

bash

export AWS_ACCESS_KEY_ID=…
export AWS_SECRET_ACCESS_KEY=…
export AWS_REGION=us-east-1

No code changes are needed to move from MinIO to production. Set the variables and change the URL.

Backends in JavaScript

JavaScript takes a backend object rather than a URL string. Only get is required, and a get-only backend is a read-only backend — the write entry points check for put up front and refuse by name:

interface Backend {
  get(path: string, range?: { start: number; end: number })
    : Promise<Uint8Array | { bytes: Uint8Array; etag?: string; totalLength?: number }>
  getStream?(path: string)
    : Promise<ReadableStream<Uint8Array> | {
        stream: ReadableStream<Uint8Array>;
        etag?: string;
        totalLength?: number;
      }>
  head?(path: string): Promise<{ exists?: boolean; etag?: string; size?: number }>
  put?(path: string, bytes: Uint8Array,
       opts?: { ifMatch?: string; ifNoneMatchStar?: boolean })
    : Promise<{ status: "created" | "exists" | "casFailed"; etag?: string }>
  delete?(path: string): Promise<void>
  list?(prefix: string): Promise<string[]>
}

Passing null instead uses plain fetch against the URI's base, which is enough to read a public bucket:

range is half-open — [start, end), the shape the Rust core uses. An HTTP Range header is inclusive, so a fetch-based backend must send bytes=${start}-${end - 1}. Sending end fetches one byte too many.

getStream is optional. Existing backends remain readable through get; implementing a bounded stream lets DreamDB scan historical oversized inline Tracks without first aggregating the complete object in wasm memory. A structured get or getStream response should include totalLength when known so range verification can distinguish the returned slice from the complete object.

const space = await Space.fromUri("https://bucket.s3.amazonaws.com/refs/photos", null);

The bundled S3Backend is read-only. For server-side writes, a fetch wrapper of about twenty lines is enough — get with a Range header, head, and put translating ifMatch to If-Match and ifNoneMatchStar to If-None-Match: *, returning casFailed on a 412.

Writing from a browser

Credentials must not reach the page. PresignedBackend keeps them out: your server mints short-lived presigned PUT URLs, and performs the ref's compare-and-swap itself.

import { Writer, PresignedBackend } from "@dreamlake/dreamdb";

const backend = new PresignedBackend({
  readBase: "https://bucket.s3.amazonaws.com",
  mintPut: async (paths) => (await postJson("/api/dreamdb/sign", { paths })).urls,
  commitRef: async (path, opts, bytes) =>
    await postJson("/api/dreamdb/ref", { path, bytesBase64: base64(bytes), ...opts }),
});

const w = await Writer.open("https://bucket.s3.amazonaws.com/refs/photos", backend);

Anchors are nanoseconds, and in JavaScript you pass a bigint. A number is accepted only below Number.MAX_SAFE_INTEGER; above that the SDK throws instead of rounding.

Two rules for anything in front of a bucket

Range should be honoured. DreamDB reads exact vectors by byte offset, and asks for half-open [start, end) ranges that the HTTP layer sends as inclusive bytes=start-(end-1). A proxy or server that ignores Range and replies 200 with the whole object does not break correctness — the connectors detect it and slice locally — but every ranged read then transfers the whole object, which makes large datasets impractical. Such a backend also fails Range conformance and cannot claim the corresponding performance profile.

CORS must expose ETag. Refs advance by compare-and-swap, so the SDK needs to read the ref's current ETag. Without ExposeHeaders, the browser hides that header, the SDK falls back to create-only, and the second commit to any ref fails while the first succeeded:

json

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://your-app.example"],
      "AllowedMethods": ["GET", "HEAD", "PUT"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag", "Content-Range", "Content-Length"],
      "MaxAgeSeconds": 3600
    }
  ]
}

Layout on disk

Whatever the backend, the object layout is the same, which is why a directory served over HTTP is indistinguishable from a bucket:

<root>/
├── refs/<name>            # the pointer a ref name resolves to
├── manifests/<hash>       # immutable versions, each naming its parent
├── genesis                # the timeline root
├── spatial-index/         # trained indexes
├── scalar-index/
└── <timeline>/…           # track data, addressed by content

Nothing here is a database file. Every object is content-addressed and immutable except refs/<name>, which is a single pointer updated with compare-and-swap.