# DreamDB — Full documentation > DreamDB is a storage and retrieval protocol for multimodal signals anchored to a shared timeline — append-only, content-addressed, and searchable by semantic features. Generated from https://dreamdb.dreamlake.ai. 57 pages. --- Source: https://dreamdb.dreamlake.ai # DreamDB > A searchable, versioned, distributed memory protocol for the multimodal information of human civilization. DreamDB is a **file and wire protocol** for multimodal data — a storage and retrieval specification that any compliant backend (object store, distributed FS, content-addressed network) can implement. The spec describes how multimodal signals (video, audio, text, vectors) are anchored to a shared timeline, written immutably, addressed by their semantic features, and consumed as native streams. ## Four first principles 1. **Time is the sole primary key.** Anchor everything to a high-precision timeline. Abolish human-defined IDs. 2. **Immutability is the bedrock of collaboration.** Append-only, content-addressed. New information is layered on, never overwritten. 3. **Retrieval is probabilistic localization, not scanning.** Vector features encode directly into storage paths. Search is a coordinate calculation, not a traversal. 4. **Data is stream.** Encapsulation is streaming-native. The index *is* the player's seek pointer. ## What you get DreamDB is a Rust reference implementation of the protocol (21 spec docs, ~9000 lines) plus a Python SDK that exposes it as a multimodal versioned dataset: ```python import dreamdb_dataset as vd # Define a schema with image + CLIP embedding + label (chainable) schema = (vd.Schema() .add_image("image", mime="jpeg") .add_embedding("embedding", dim=512, algorithm="dreamdb.ivf-cosine", rerank=True) .add_scalar_categorical("label")) # (Or imperative if you prefer: schema = vd.Schema(); schema.add_image(...); ...) # Create a Dataset on MinIO/S3 backend = "http://minio:9000/imagenet-1k" ds = vd.Dataset.create("imagenet-1k", schema, backend=backend) # Append a batch of samples (one dict per row, keyed by field name) ds.append_many([ {"image": jpeg_bytes, "embedding": clip_vec, "label": "tabby cat"}, # … one dict per sample ]) # Snapshot the dataset at a point in time (for reproducible training) v = ds.snapshot("baseline-2026-05-18") # Query: top-K nearest neighbors via spatial indexing hits = ds.iter_vector(field="embedding", query=clip_embed("a brown dog"), top_k=10) # Sharded parallel ingest across N workers, then merge back # (each worker calls ds.branch("worker-N") + append_many on its slice) ds.merge_many(["worker-0", "worker-1", "worker-2"]) # GDPR-style deletion (read-side suppression; storage compaction is a separate op) ds.delete(anchors=[1779083474791115000], reason="gdpr") # Time-travel: open any prior snapshot old = vd.Dataset.open_at(v, backend=backend) ``` The same protocol surface is available natively in Rust (`dreamdb-dataset` crate), via CLI (`dreamdb` binary for maintenance operations), and inside a browser (a JS implementation in `dreamdb-dataset-python/python/dreamdb_dataset/web/` for client-side search demos). ## Status (2026-05-18) | Component | Status | Notes | |---|---|---| | Protocol spec (specs 0000–0020) | ✅ v0 complete | 21 docs, ~9000 lines. Multi-parent fused-merge formalized in `spec/0008 §5.3` | | Rust SDK (`dreamdb-dataset`) | ✅ shipping | Dataset/Schema/Sample, ingest, iter_stream, snapshot/branch/merge, delete, tombstones | | Python bindings | ✅ shipping | PyO3 wheel; PyTorch IterableDataset + Arrow batches | | CLI (`dreamdb` binary) | ✅ shipping | ada-ivf-step (k8s 4-stage), gc (k8s-shardable), delete, merge-many, inspect, snapshot | | HTTP connector (MinIO/S3) | ✅ shipping | `dreamdb-connector-http`; conformance suite passes | | Browser SDK + UI (`browse.html`) | ✅ shipping | Time-travel viewer, semantic search, IVF+RaBitQ on the client | | Conformance test vectors | ⚠️ partial | CBOR / address / time / spatial covered (`dreamdb-conformance/`); cross-SDK interop deferred | | Billion-scale benchmark | ⏳ in progress | imagenet-1k (1.3M records) ingest running 2026-05-18; SIFT1M / LAION-100M next | | Test suite | ✅ 721 green | Across all crates; 0 failed | | Production hardening | ⚠️ pre-1.0 | Observability spec, RBAC, language-binding stability deferred to v0.1+ | ## Backends The HTTP connector talks to any S3-shape endpoint. Choose one: | Backend | Setup | Auth | |---|---|---| | **MinIO (local dev)** | `docker run ... minio/minio` + `mc anonymous set public` | None — unsigned requests | | **AWS S3** | Create bucket; set `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION` env vars | SigV4 (auto-detected from env) | | **Cloudflare R2 / Backblaze B2 / Wasabi** | Create bucket + S3-compat API token; same env vars as AWS | SigV4 (auto-detected) | | **`memory://`** | Built-in in-process backend | None (testing only) | The connector detects `AWS_*` env vars and auto-signs with SigV4 when present. No code changes needed to migrate from local MinIO to production S3 — set the env vars and point at the new backend URL. ## 60-second quickstart Requires Rust ≥1.83, Python ≥3.10, Docker (for MinIO), and `uv` (or any pip-equivalent). ```bash # 1. Clone + build git clone https://github.com//dreamdb cd dreamdb cargo build --release # ~5 min cold; <30s incremental # 2. Start a MinIO backend docker run -d --name dreamdb-minio \ -p 9000:9000 -p 9001:9001 \ -e MINIO_ROOT_USER=dreamdb \ -e MINIO_ROOT_PASSWORD=dreamdbsecret \ minio/minio:latest server /data --console-address ":9001" docker exec dreamdb-minio mc alias set local http://localhost:9000 dreamdb dreamdbsecret docker exec dreamdb-minio mc mb local/demo docker exec dreamdb-minio mc anonymous set public local/demo # 3. Build + install the Python wheel cd dreamdb-dataset-python uv pip install maturin maturin develop --release # 4. A 30-line "hello world" — write 1000 random vectors, query nearest ``` *Run with `uv run hello_world.py`* ```python #!/usr/bin/env -S uv run # /// script # dependencies = ["numpy", "dreamdb-dataset"] # /// import numpy as np, dreamdb_dataset as vd s = vd.Schema().add_embedding("v", dim=32, algorithm="dreamdb.lsh-cosine") ds = vd.Dataset.create("hello", s, backend="http://localhost:9000/demo") samples = [{"v": np.random.randn(32).astype(np.float32)} for _ in range(1000)] ds.append_many(samples) q = np.random.randn(32).astype(np.float32).tolist() batches = ds.iter_vector(field="v", query=q, top_k=5) for batch in batches: for anchor in batch["_time_anchors"]: print(f"hit: anchor={anchor}") ``` ```bash # 5. Inspect the Manifest history ./target/release/dreamdb inspect \ --backend http://localhost:9000/demo --ref-name hello --max-depth 10 ``` A 10-minute hands-on walkthrough — schema + ingest + snapshot + query + PyTorch DataLoader + time-travel + delete + sharded ingest — lives in [`docs/tutorial.md`](docs/tutorial.md). For a substantial real-data ingest at scale, see [`dreamdb-dataset-python/examples/ingest_imagenet100_clip.py`](dreamdb-dataset-python/examples/ingest_imagenet100_clip.py). ## Repository layout ``` dreamdb/ ├── spec/ # 21 protocol spec docs (start with spec/0000) ├── INDEX.md # spec navigation + Open Questions audit ├── design/ # implementation design docs (operator-facing) ├── dreamdb-core/ # addressing, hashing, CBOR primitives — shared by every layer ├── dreamdb-protocol/ # Object types (Manifest, Track, Bucket, ...) + verbs (Open/Append/Query/Stream) ├── dreamdb-connector/ # storage trait + memory backend ├── dreamdb-connector-http/ # MinIO/S3 implementation ├── dreamdb-dataset/ # high-level SDK (Schema, Sample, Dataset, Filter, Batch, MergeStrategy) ├── dreamdb-dataset-python/ # PyO3 bindings + Python wrapper + browser JS demo ├── dreamdb-cli/ # operator binaries (rebuild-ivf, ada-ivf-step, gc, delete, merge-many) ├── dreamdb-conformance/ # JSON test vectors per spec/0009 §3 └── dreamdb-bench/ # microbenchmarks (storage / latency / recall) ``` ## Spec roadmap The protocol spec is in [`spec/`](spec/). Start with [`spec/0000-overview.md`](spec/0000-overview.md), then read in order — each doc inherits vocabulary and decisions from the ones before it. [`INDEX.md`](INDEX.md) is a navigation index with the full Open Questions audit. **What's covered (21 specs, v0 + Phase-3 + Phase-4 drafts)**: data model, content-addressing, time encoding, spatial indexing (LSH/IVF/IMI), backend interface, eight protocol verbs, streaming encapsulation, versioning + multi-parent merge, conformance, vector compression (PQ/RaBitQ), scalar indexing, federation, graph indexing (Vamana), streaming extensions, hybrid retrieval, streaming freshness, schema evolution, multi-tenant, encryption, tombstones. **Honest gaps (deferred to v0.1+)**: - **Observability**: no spec for metrics / structured logging / health checks. SDKs log at debug/info but the surface isn't standardized. - **Cross-SDK interop**: spec/0009 conformance covers protocol-level vectors but not "Rust ingester + Python reader" round-trips. - **Security model**: spec/0019 covers encryption at rest; RBAC / capability tokens / audit logging are sketched in spec/0012 §5 but not standalone. - **Tombstone compaction**: tombstones suppress on read (spec/0020 §5); the storage-reclamation operator is deferred. - **Chinese translations** ([`spec/chn/`](spec/chn/)): cover specs 0000–0009 only; later specs not translated. ## Reading order For protocol understanding: `spec/0000` → `0001` → `0002` → `0004` → `0007` → `0008` → `0006` (read these in this order, not numerical order). For implementation reading: `dreamdb-core/src/address.rs` → `dreamdb-protocol/src/manifest.rs` → `dreamdb-dataset/src/dataset.rs` → `dreamdb-dataset/src/dataset/append.rs`. For operator workflows: `design/0006-10b-scale-blockers.md` → `design/0007-sharded-ingest.md`. ## License Apache-2.0 / MIT dual. --- Source: https://dreamdb.dreamlake.ai/installation # Installation DreamDB ships as a single npm package for both Node.js and browser environments. ## npm / pnpm / yarn ```bash npm install @dreamlake/dreamdb ``` ```bash pnpm add @dreamlake/dreamdb ``` ```bash yarn add @dreamlake/dreamdb ``` Then import the SDK in your application: ```ts ``` ## Browser via CDN For quick prototypes or standalone HTML pages, load DreamDB from a CDN: ```html ``` The script exposes a global `DreamDB` object: ```html ``` ## Requirements | Environment | Minimum version | Notes | |---|---|---| | **Node.js** | 18+ | Uses the native `fetch` API (stable since Node 18) | | **Browser** | Any modern browser | Chrome 66+, Firefox 57+, Safari 12.1+, Edge 79+ | | **TypeScript** | 5.0+ | Type declarations are bundled in the package | The only runtime dependency is a standards-compliant `fetch` implementation. No native add-ons or WebAssembly modules are required. ## Backend setup DreamDB stores data on any S3-compatible object store. For local development, the fastest path is MinIO: ```bash docker run -d --name dreamdb-minio \ -p 9000:9000 -p 9001:9001 \ -e MINIO_ROOT_USER=dreamdb \ -e MINIO_ROOT_PASSWORD=dreamdbsecret \ minio/minio:latest server /data --console-address ":9001" # Create a bucket and make it publicly readable. docker exec dreamdb-minio mc alias set local http://localhost:9000 dreamdb dreamdbsecret docker exec dreamdb-minio mc mb local/demo docker exec dreamdb-minio mc anonymous set public local/demo ``` For production, point at any S3-compatible endpoint (AWS S3, Cloudflare R2, Backblaze B2, Wasabi). Set the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION` environment variables and the HTTP connector auto-signs requests with SigV4. ## Install the docs as a skill These docs ship as an [Agent Skill](/llm-readable.md) so your agent can answer DreamDB questions accurately — without you pasting context. Install it once and Claude loads it on demand. **Claude Code — this project only** (drop it in the project's skills dir): ```bash curl -L https://dreamdb.dreamlake.ai/skills/dreamdb.zip -o dreamdb.zip unzip dreamdb.zip -d .claude/skills/ && rm dreamdb.zip ``` **Claude Code — every project** (install under your home config): ```bash curl -L https://dreamdb.dreamlake.ai/skills/dreamdb.zip -o dreamdb.zip unzip dreamdb.zip -d ~/.claude/skills/ && rm dreamdb.zip ``` The skill is `dreamdb/` with a `SKILL.md` and one markdown reference file per docs page. It's regenerated on every deploy, so it never drifts from the site. > **Note:** **No install needed for one-off questions.** Point any agent at > [`https://dreamdb.dreamlake.ai/llms.txt`](https://dreamdb.dreamlake.ai/llms.txt) > (an index) or `https://dreamdb.dreamlake.ai/llms-full.txt` (the whole site in one > file). See [LLM-Readable Docs](/llm-readable.md) for all the ways to consume these > docs as markdown. ## Next steps - [Quick Start](/tutorial.md) -- a 10-minute end-to-end walkthrough with ingest, query, and time-travel. - [TypeScript SDK Reference](/typescript-sdk.md) -- the full API surface for the `@dreamlake/dreamdb` package. - [Browser Demo](/browser-demo.md) -- try DreamDB in the browser without installing anything. --- Source: https://dreamdb.dreamlake.ai/tutorial # DreamDB in 10 minutes A hands-on walkthrough that takes you from `git clone` to "I just ran semantic search over my own data, then deleted a record, then time-traveled back to before the deletion." Runnable end-to-end on a single machine. We'll build a tiny dataset of 200 images + CLIP embeddings + labels, then: - snapshot the dataset for reproducibility - query it semantically - iterate it as a PyTorch DataLoader - open a prior snapshot (time-travel) - delete a record (tombstone) - view the history visually in a browser - run operator maintenance (`ada-ivf-status`, `gc`) - shard ingest across two workers and merge them back By the end you'll have used every primitive in `spec/0006`'s eight-verb taxonomy and every operator workflow in `design/0006-10b-scale-blockers.md`. ## 0. Prerequisites - **Rust ≥ 1.83**, **Python ≥ 3.10**, **Docker**, and [**`uv`**](https://github.com/astral-sh/uv) (or any pip-equivalent). - ~10 GB free disk for MinIO + the wheel. The tutorial uses CLIP-ViT-B/32 for embeddings. CPU works; GPU (MPS on Apple Silicon, CUDA otherwise) makes step 3 ~5× faster. ## 1. Clone and build ```bash git clone https://github.com//dreamdb cd dreamdb cargo build --release # ~5 min cold; <30s incremental ``` The `target/release/dreamdb` binary will exist when this finishes. ## 2. Start MinIO ```bash docker run -d --name dreamdb-tutorial-minio \ -p 9000:9000 -p 9001:9001 \ -e MINIO_ROOT_USER=dreamdb \ -e MINIO_ROOT_PASSWORD=dreamdbsecret \ minio/minio:latest server /data --console-address ":9001" # Set up the CLI client + create the tutorial bucket (public for simplicity). docker exec dreamdb-tutorial-minio mc alias set local http://localhost:9000 dreamdb dreamdbsecret docker exec dreamdb-tutorial-minio mc mb local/tutorial docker exec dreamdb-tutorial-minio mc anonymous set public local/tutorial ``` Backend URL throughout this tutorial: `http://localhost:9000/tutorial`. ## 3. Install the Python wheel ```bash cd dreamdb-dataset-python uv pip install maturin maturin develop --release # ~2 min; builds the PyO3 wheel ``` After this, `import dreamdb_dataset` works in any Python ≥3.10 environment that uses this venv. ## 4. Hello world — 1000 random vectors The smallest end-to-end DreamDB program. Schema with one embedding field, 1000 random samples, a top-K query. *Run with `uv run hello_world.py`* ```python #!/usr/bin/env -S uv run # /// script # dependencies = ["numpy", "dreamdb-dataset"] # /// import numpy as np import dreamdb_dataset as vd BACKEND = "http://localhost:9000/tutorial" # 4a. Define a schema and create the Dataset. schema = vd.Schema().add_embedding("v", dim=32, algorithm="dreamdb.lsh-cosine") ds = vd.Dataset.create("hello", schema, backend=BACKEND) # 4b. Append 1000 random unit vectors. Samples are plain dicts. rng = np.random.default_rng(42) samples = [{"v": rng.standard_normal(32).astype(np.float32)} for _ in range(1000)] ds.append_many(samples) # 4c. Query the top 5 nearest neighbors to a fresh random vector. q = rng.standard_normal(32).astype(np.float32).tolist() batches = ds.iter_vector(field="v", query=q, top_k=5) for batch in batches: for anchor in batch["_time_anchors"]: print(f" hit: anchor={anchor}") ``` Output looks like: ``` hit: anchor=1779090000000000001 hit: anchor=1779090000000000123 hit: anchor=1779090000000000456 hit: anchor=1779090000000000789 hit: anchor=1779090000000000999 ``` Anchors are 64-bit time-anchors (the "T" in spec/0003 — DreamDB's only primary key). ## 5. A real multimodal dataset Now with images, CLIP embeddings, and labels. We'll use 200 sample images bundled with `dreamdb-dataset-python/examples/sample_images/` (download a handful manually or substitute any 200 JPEGs — the tutorial works with any small set). *Run with `uv run ingest_clip.py`* ```python #!/usr/bin/env -S uv run # /// script # dependencies = ["numpy", "torch", "open-clip-torch", "pillow", "dreamdb-dataset"] # /// import io, glob, numpy as np, dreamdb_dataset as vd import open_clip, torch from PIL import Image BACKEND = "http://localhost:9000/tutorial" device = "mps" if torch.backends.mps.is_available() else ("cuda" if torch.cuda.is_available() else "cpu") # 5a. Load CLIP-ViT-B/32. ~600 MB download first time. model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="openai") model = model.to(device).eval() # 5b. Schema: image (jpeg) + CLIP embedding (512-dim) + label (categorical). schema = (vd.Schema() .add_image("image", mime="jpeg") .add_embedding("embedding", dim=512, algorithm="dreamdb.lsh-cosine") .add_scalar_categorical("label")) ds = vd.Dataset.create("multimodal", schema, backend=BACKEND) # 5c. Read up to 200 JPEGs from any folder you have. (Adjust path.) paths = sorted(glob.glob("/Users/me/Pictures/*.jpg"))[:200] print(f"found {len(paths)} images") samples = [] clip_imgs = [] for p in paths: with open(p, "rb") as f: jpeg = f.read() img = Image.open(io.BytesIO(jpeg)).convert("RGB") clip_imgs.append(preprocess(img)) samples.append({"image": jpeg, "label": p.split("/")[-1]}) # 5d. CLIP-encode in one batch and attach embeddings. with torch.no_grad(): feats = model.encode_image(torch.stack(clip_imgs).to(device)) feats = feats / feats.norm(dim=-1, keepdim=True) feats = feats.cpu().float().numpy() for s, f in zip(samples, feats): s["embedding"] = f # 5e. Append. One round-trip per modality, regardless of batch size. ds.append_many(samples) print(f"appended {len(samples)} samples; manifest hash = {ds.current_manifest()}") ``` ## 6. Snapshot the dataset Snapshots are immutable named labels pinning a Manifest hash. Use them for "training run X used dataset state Y" reproducibility. ```python import dreamdb_dataset as vd ds = vd.Dataset.open_ref("multimodal", backend="http://localhost:9000/tutorial") v = ds.snapshot("baseline-200") print(f"snapshot label = {v['label']}") print(f"snapshot manifest = {v['manifest']}") print(f"timeline = {v['timeline']}") ``` A snapshot is just a Ref pointing at a specific Manifest. No data copy — the 200-sample dataset is now reachable at TWO refs: `multimodal` (the live working ref) and `baseline-200` (frozen). They share storage. ## 7. Query semantically ```python #!/usr/bin/env -S uv run # /// script # dependencies = ["torch", "open-clip-torch", "dreamdb-dataset"] # /// import open_clip, torch, dreamdb_dataset as vd # Encode a text query with CLIP's text tower. model, _, _ = open_clip.create_model_and_transforms("ViT-B-32", pretrained="openai") tokenizer = open_clip.get_tokenizer("ViT-B-32") with torch.no_grad(): q = model.encode_text(tokenizer(["a sunset"])) q = (q / q.norm(dim=-1, keepdim=True))[0].numpy() ds = vd.Dataset.open_ref("multimodal", backend="http://localhost:9000/tutorial") batches = ds.iter_vector(field="embedding", query=q.tolist(), top_k=10) for batch in batches: for anchor, label in zip(batch["_time_anchors"], batch["label"]): print(f" hit: anchor={anchor} label={label}") ``` CLIP image and text embeddings share a space, so a text query finds visually-matching images. ## 8. Iterate as a PyTorch DataLoader ```python #!/usr/bin/env -S uv run # /// script # dependencies = ["torch", "pyarrow", "numpy", "dreamdb-dataset"] # /// import dreamdb_dataset as vd from dreamdb_dataset.torch import DreamDBIterableDataset from torch.utils.data import DataLoader ds = vd.Dataset.open_ref("multimodal", backend="http://localhost:9000/tutorial") torch_ds = DreamDBIterableDataset(ds, batch_size=32, fields=["embedding", "label"]) loader = DataLoader(torch_ds, batch_size=None, num_workers=0) for batch in loader: # `batch` is a pyarrow.RecordBatch with one column per requested field # plus `_anchor`. Convert as needed for your training loop. embeddings = batch.column("embedding").to_numpy(zero_copy_only=False) print(f"batch shape: {embeddings.shape}") break ``` `DreamDBIterableDataset` is a `torch.utils.data.IterableDataset` over the dataset's Arrow batches. Multi-worker support partitions batches by index across workers. ## 9. Time-travel — open a prior snapshot ```python import dreamdb_dataset as vd # Open the dataset at the "baseline-200" snapshot specifically. v = {"label": "baseline-200", "manifest": None, "timeline": None} # (You can also pass the dict returned by `ds.snapshot(...)` directly.) old = vd.Dataset.open_ref("baseline-200", backend="http://localhost:9000/tutorial") print(f"manifest at baseline-200: {old.current_manifest()}") ``` The "live" `multimodal` ref might have moved on (more appends, ada-ivf rebuilds, deletes), but the `baseline-200` snapshot is byte-identical to what was there at snapshot time — every Object it transitively references is content-addressed and immutable. ## 10. Visualize in a browser ```bash # Serve the static browser UI (no build step — it's plain JS). cd dreamdb-dataset-python/examples/web python -m http.server 8080 ``` Open `http://localhost:8080/browse.html` in Chrome. Point it at the backend URL `http://localhost:9000/tutorial` and the ref name `multimodal`. You'll see: - A scrollable grid of the 200 images you appended. - A search box that runs semantic queries against your CLIP embeddings, client-side. - A "⏳ history" button that lists every Manifest in the ref's history; click one to re-render the entire view at that point in time. (This is the "killer feature" — the same dataset, at any past moment, with one click.) ## 11. Operator maintenance Two single-machine ops every DreamDB operator runs. ### 11a. Index health ```bash ./target/release/dreamdb ada-ivf-status \ --backend http://localhost:9000/tutorial \ --timeline \ --modality embedding.f32.dim=512.bucketed ``` Reports per-cell record counts, the imbalance score, and a MONITOR / SPLIT / MERGE recommendation. At 200 records this will be MONITOR (k=√200 ≈ 14 cells, ~14 records each, nothing to do). ### 11b. Garbage collection ```bash ./target/release/dreamdb gc \ --backend http://localhost:9000/tutorial \ --keep-manifests 100 \ --keep-since 24h \ --dry-run ``` Reports how many Objects would be reclaimed. At billion-scale, run with `--shard N --of M` across multiple k8s pods. ## 12. Delete a record (tombstone) GDPR-style suppression: read paths skip the named anchor; storage compaction is a separate operator pass (deferred per `spec/0020 §6`). ```bash # Pick an anchor from step 7's query output — e.g. 1779090000000000456. ./target/release/dreamdb delete \ --backend http://localhost:9000/tutorial \ --ref-name multimodal \ --reason gdpr \ 1779090000000000456 ``` Or from Python: ```python ds = vd.Dataset.open_ref("multimodal", backend=BACKEND) new_hash = ds.delete([1779090000000000456], reason="gdpr") print("tombstone set now:", ds.tombstone_set()) ``` Now re-run the semantic query from step 7. The deleted anchor is gone. But re-run it against the `baseline-200` snapshot — the record is still there, because that snapshot pins the Manifest from before the delete: ```python old = vd.Dataset.open_ref("baseline-200", backend=BACKEND) batches = old.iter_vector(field="embedding", query=q.tolist(), top_k=10) # the deleted anchor still appears here ``` This is DreamDB's time-travel property in action: deletion doesn't rewrite history, it adds a Manifest with a `dreamdb.tombstones` registry entry. Old snapshots that don't reference the tombstone see the original data. ## 13. Sharded ingest with `merge-many` For real-scale ingest, run N workers each ingesting into their own branch, then merge them all into trunk in one shot. Worker 0 -- branch and append its slice: ```python #!/usr/bin/env -S uv run # /// script # dependencies = ["numpy", "dreamdb-dataset"] # /// import numpy as np, dreamdb_dataset as vd ds = vd.Dataset.open_ref("multimodal", backend="http://localhost:9000/tutorial") w = ds.branch("ingest-w-0") rng = np.random.default_rng(0) w.append_many([{"image": b"fake-jpeg-0", "embedding": rng.standard_normal(512).astype(np.float32), "label": f"w0-{i}"} for i in range(50)]) ``` Worker 1 -- same, different slice: ```python #!/usr/bin/env -S uv run # /// script # dependencies = ["numpy", "dreamdb-dataset"] # /// import numpy as np, dreamdb_dataset as vd ds = vd.Dataset.open_ref("multimodal", backend="http://localhost:9000/tutorial") w = ds.branch("ingest-w-1") rng = np.random.default_rng(1) w.append_many([{"image": b"fake-jpeg-1", "embedding": rng.standard_normal(512).astype(np.float32), "label": f"w1-{i}"} for i in range(50)]) ``` Orchestrator merges both branches into trunk in one shot -- either via CLI: ```bash ./target/release/dreamdb merge-many \ --backend http://localhost:9000/tutorial \ --ref-name multimodal \ ingest-w-0 ingest-w-1 ``` ...or from Python: ```python import dreamdb_dataset as vd trunk = vd.Dataset.open_ref("multimodal", backend="http://localhost:9000/tutorial") final_manifest = trunk.merge_many(["ingest-w-0", "ingest-w-1"]) print(f"new trunk manifest: {final_manifest}") ``` The orchestrator publishes a new Manifest with `parents = [trunk_tip, w0_tip, w1_tip]` (chained sequentially in v0). At 10B scale you'd run 64 workers in parallel against a k8s `Job`; the merge-many step takes ~minutes regardless. See `design/0007-sharded-ingest.md` for the algorithm and the k8s YAML pattern. ## 14. Moving to S3 (optional) When local MinIO outgrows your laptop disk, the path to production is just env vars — no code changes: ```bash export AWS_ACCESS_KEY_ID="AKIA…" export AWS_SECRET_ACCESS_KEY="…" export AWS_REGION="us-east-1" # Same Python / CLI commands as above; just point at the S3 bucket. python -c 'import dreamdb_dataset as vd; print(vd.Dataset.open_ref("multimodal", backend="https://s3.us-east-1.amazonaws.com/my-bucket").count())' ``` The connector auto-detects the env vars and signs every request with SigV4. Works against AWS S3, Cloudflare R2, Backblaze B2, Wasabi, and any other S3-compatible endpoint. For R2, set `AWS_REGION=auto` and use your R2 endpoint URL. ## 15. What now? You've used the full DreamDB surface. Here's where to read next: - **For the protocol**: `spec/0000-overview.md` is the entry point; `INDEX.md` is the navigation index for all 21 specs. - **For real ingest at scale**: `design/0006-10b-scale-blockers.md` (everything that makes DreamDB 10B-ready). - **For sharded ingest mechanics**: `design/0007-sharded-ingest.md`. - **For the Rust SDK**: start at `dreamdb-dataset/src/dataset.rs`; the public surface is small and well-documented. - **For operator workflows**: `dreamdb --help` lists every CLI verb. ## Cleanup ```bash docker rm -f dreamdb-tutorial-minio docker volume rm $(docker volume ls -q | grep minio) 2>/dev/null || true ``` The tutorial bucket and all 200 samples + 1000 random vectors are gone. DreamDB itself is just files on disk; nothing else persists. ## Status notes This tutorial reflects the v0 reference implementation as of 2026-05-18. Two things to know: - **`iter_arrow_batches`** materializes the whole dataset into memory before yielding. For 10B-scale Python streaming, use `Dataset.iter_stream(batch_size, fields)` instead — it returns a true generator (`StreamBatchIter`) backed by a Rust mpsc channel, with bounded per-batch RAM. Embedding-only in v0; multi-modality merge-join is the next streaming-iter slice. - **CLIP-ViT-B/32 on M-series MPS** runs at ~200-300 samples/sec — that's the GPU forward-pass limit, not a DreamDB bottleneck. Production ingest uses sharded workers (step 13). --- Source: https://dreamdb.dreamlake.ai/browser-demo # Browser Demo DreamDB includes a fully client-side dataset explorer that runs entirely in the browser. No application server is needed -- the page talks directly to the S3-compatible backend over HTTP. **Live demo:** [demo.dreamdb.dreamlake.ai](https://demo.dreamdb.dreamlake.ai) ## What the demo does The browser demo is a single-page application that renders a DreamDB Space URI into an interactive explorer. Given a backend URL and a ref name, it resolves the Manifest, walks every Track, and presents: - **Dataset explorer** -- a card grid showing every sample in the dataset. Each card displays the image thumbnail (or video player), scalar field values (labels, splits, timestamps), and the sample's time anchor. - **Semantic search** -- type a natural-language query and the demo encodes it with CLIP (ViT-B/32, running client-side via `transformers.js`) and performs a vector search against the dataset's embedding index. Results are ranked by cosine similarity. - **Time-travel** -- the history panel lists every Manifest in the ref's parent chain. Click any entry to re-render the entire view at that point in time. This is the same immutable Object graph that powers the Python SDK's `Dataset.open_at()`. - **Video playback** -- video fields render inline with HLS streaming support for chunked items. - **Point cloud viewer** -- 3D LiDAR and scene-update tracks render in an interactive Three.js viewport with orbit controls. ## Connecting your own data ### Point at a local MinIO instance If you followed the [Quick Start](/tutorial.md) and have MinIO running on `localhost:9000`, open the demo and enter: - **Backend URL:** `http://localhost:9000/tutorial` - **Ref name:** `multimodal` The demo loads the Manifest, resolves all Tracks, and renders the samples. ### Add your own S3 bucket To browse a dataset on AWS S3 or another hosted backend: 1. Ensure the bucket allows public read access (or configure CORS to permit browser requests from the demo origin). 2. Enter the full backend URL, e.g. `https://s3.us-east-1.amazonaws.com/my-bucket`. 3. Enter the ref name of the dataset you want to browse. The demo stores your backend list in `localStorage` so you do not need to re-enter it on each visit. ### CORS configuration For cross-origin access from the browser, the S3 bucket must return appropriate CORS headers. A minimal MinIO policy: ```bash docker exec dreamdb-minio mc anonymous set public local/my-bucket ``` For AWS S3, add a CORS configuration to the bucket: ```json { "CORSRules": [ { "AllowedOrigins": ["*"], "AllowedMethods": ["GET", "HEAD"], "AllowedHeaders": ["*"], "MaxAgeSeconds": 3600 } ] } ``` ## Features in detail ### Dataset structure panel The top of the page shows the dataset's structural metadata: Manifest hash, timeline ID, writer identity, and a badge for each Track (image, video, embedding, scalar). Badges show the item count per track and distinguish fragment, spatial, and scalar tracks by color. ### Semantic search The search bar loads the CLIP text encoder (ViT-B/32) on first use via [transformers.js](https://huggingface.co/docs/transformers.js). The text query is encoded into a 512-dimensional vector and matched against the dataset's embedding index using DreamDB's IVF + RaBitQ spatial search -- all running in the browser, no server round-trip. Search results update the grid in ranked order. Each card shows its cosine similarity score. ### Time-travel history Click the history button to see every Manifest in the dataset's parent DAG, ordered newest to oldest. Each entry shows: - Manifest hash (base32) - Writer-asserted timestamp - Track count and modality summary - Parent count (0 for genesis, 1 for normal appends, 2+ for merges) Click any history entry to re-render the entire explorer at that prior state. Because every Object is content-addressed and immutable, historical views are as fast and reliable as the current one. ### Time-range queries Select a time window to filter samples to a specific range of anchors. The demo uses `DreamDBSpace.itemsInRange()` to restrict the view without re-fetching every Track. ## Running the demo locally The demo is a static HTML page with no build step: ```bash cd demo python -m http.server 8080 ``` Open `http://localhost:8080/browse.html` in a modern browser. The page loads `dreamdb.js` (the DreamDB browser SDK) and `viz.js` (the Three.js-based point cloud renderer) as plain script tags. ## Architecture The browser SDK (`dreamdb.js`) implements the same protocol primitives as the Rust and Python SDKs: - **Multihash + base32** addressing for content-addressed Objects - **CBOR decoding** for Manifests, Tracks, and Buckets - **Space URI parsing** to resolve refs and manifests from any S3 endpoint - **Concurrent fetch** with bounded parallelism (32 in-flight requests) for Track and Bucket resolution No WebAssembly or native modules are involved. The entire SDK is a single JavaScript file that runs in any browser with `fetch` and `DataView` support. --- Source: https://dreamdb.dreamlake.ai/llm-readable # LLM-Readable Docs These docs are built to be read by agents as easily as by people. Every page has a markdown twin, and the whole corpus is published in the formats LLM tooling already looks for — so you can point Claude (or any agent) at DreamDB and have it answer accurately. ## Fetch a single page Append `.md` to any docs URL to get the raw markdown — no nav, no chrome: ```bash curl https://dreamdb.dreamlake.ai/installation.md ``` Every page also advertises its markdown twin in the HTML head: ```html ``` ## The whole site, two ways - **[`/llms.txt`](https://dreamdb.dreamlake.ai/llms.txt)** — a short, linked index of every page ([llmstxt.org](https://llmstxt.org) standard). The entry point an agent reads first to decide what to fetch. - **[`/llms-full.txt`](https://dreamdb.dreamlake.ai/llms-full.txt)** — every page concatenated into one markdown file. Drop the entire product into a context window in a single request. ## Import it as a skill The docs are also packaged as an [Agent Skill](https://dreamdb.dreamlake.ai/skills/dreamdb.zip) — a `SKILL.md` plus one markdown reference file per page. Install it so your agent loads DreamDB knowledge on demand: ```bash # Claude Code: drop it into your project (or ~/.claude) skills directory curl -L https://dreamdb.dreamlake.ai/skills/dreamdb.zip -o dreamdb.zip unzip dreamdb.zip -d .claude/skills/ ``` > **Note:** **Always current.** Every surface above — the `.md` pages, both `llms` files, > and the skill — is generated from the same source on each deploy, so none of > them can drift from what you read on the site. --- Source: https://dreamdb.dreamlake.ai/typescript-sdk # TypeScript SDK `@dreamlake/dreamdb` is the TypeScript-first client for the DreamDB protocol. It runs in Node 20+ **and** the browser, with zero runtime dependencies, and talks directly to any S3-compatible backend — no application server in the middle. ## What's in the package | Concept | Description | |---|---| | **`Schema`** | Declarative builder for dataset fields (image, video, audio, embedding, scalar). | | **`Dataset`** | Versioned multimodal collection. Create, open, append, query, snapshot, branch, merge, compact, delete. | | **`Space`** | Read-side view of a Manifest + its Tracks. Designed for browsers, SSR, and CLIs that only consume. | | **`Backend`** | Pluggable storage. Two built-in: `S3Backend` (any S3-compatible HTTP endpoint) and `MemoryBackend` (tests). | | Query primitives | `normalizeL2`, `dotL2Normalized`, RaBitQ + PQ decoders, LSH + IMI helpers — for code that drops below the high-level API. | ## Read the docs - [Install](/typescript-sdk-install.md) — npm install + runtime requirements - [Quickstart](/typescript-sdk-quickstart.md) — your first dataset in 30 lines - [API reference](/typescript-sdk-api.md) — every public export with signatures - [Browser usage](/typescript-sdk-browser.md) — `Space.fromUri`, blob URLs, the read path - [React example](/typescript-sdk-react.md) — minimal grid of thumbnails - [Semantic search](/typescript-sdk-semantic-search.md) — CLIP text encoder + `iterVector` ## Architecture ``` ┌──────────────────┐ ┌──────────────────────┐ ┌──────────────────┐ │ Your code │───▶│ @dreamlake/dreamdb │───▶│ Object store │ │ (Node/Browser) │ │ (zero-dep client) │ │ S3 / MinIO / R2 │ └──────────────────┘ └──────────────────────┘ └──────────────────┘ HTTP only · content-addressed · spec-driven ``` Compatible with any backend that speaks S3-style HTTP GET/PUT (MinIO, AWS S3, Cloudflare R2, Wasabi, Backblaze B2). AWS SigV4 is auto-detected from the standard `AWS_*` env vars. ## Source `dreamdb-ts/` in this workspace. The package is also published standalone at [`dreamlake-ai/dreamdb-ts`](https://github.com/dreamlake-ai/dreamdb-ts). --- Source: https://dreamdb.dreamlake.ai/typescript-sdk-install # Install ```bash npm install @dreamlake/dreamdb # or pnpm / yarn / bun pnpm add @dreamlake/dreamdb ``` ## Requirements - **Node 20+** — the SDK uses native `fetch`, `BigInt`, and `Uint32Array`. - **Modern browsers** — any browser supporting ES2022 modules (Chrome 94+, Firefox 93+, Safari 15.4+). - **Zero runtime dependencies.** ## Dual entry points | Import | Use from | What's there | |---|---|---| | `@dreamlake/dreamdb` | Node, browser, SSR | Core: `Dataset`, `Schema`, `Space`, `S3Backend`, query primitives, CBOR helpers | | `@dreamlake/dreamdb/browser` | Browser only | Adds `blobUrlFor()` for `` / `