DreamDB v0 — Spec Index
A single navigable entry point for the DreamDB v0 specification. Maps every Open Question (OQ) to its resolution and points at every spec document.
The 21 Spec Documents
Read in order; each document inherits vocabulary and decisions from the ones before it.
| # | Document | What it pins down |
|---|---|---|
| 0 | spec/0000-overview.md | Vocabulary, four first principles, three-layer architecture (App → SDK → HTTP → Object Store) |
| 1 | spec/0001-data-model.md | Five entities (Space, Timeline, Track, Item, Manifest); three Track kinds; Item ≠ Object |
| 2 | spec/0002-content-addressing.md | BLAKE3, deterministic CBOR, address grammar, modality-tag grammar, paged Index Pages |
| 3 | spec/0003-time-encoding.md | u64 ns ticks, integer-only arithmetic, 16-char hex, billion-scale sizing |
| 4 | spec/0004-spatial-indexing.md | dreamdb.lsh-cosine, ChaCha20 hyperplanes, multi-table & multi-probe levers |
| 5 | spec/0005-backend-interface.md | HTTP contract, Manifest Supremacy, ETag normalization, HTTP/2 mandate |
| 6 | spec/0006-protocol-operations.md | Eight verbs, per-session cache, GC algorithm, Stream prefetch, Ref freshness |
| 7 | spec/0007-streaming-encapsulation.md | CMAF Fragments, Spatial Bucket layout, Index Page byte format, dynamic B-tree height |
| 8 | spec/0008-versioning-collab.md | Manifest DAG, branching, merging, MUST-refuse-on-incompat, monotonic-ts |
| 9 | spec/0009-conformance.md | Conformance roles, test categories, meta-OQ resolutions (CBOR tags, multihash) |
| 10 | spec/0010-vector-compression.md | VectorCompressor Object, dreamdb.qinco-cosine, modality compress= suffix, re-rank contract — draft, Phase 3 |
| 11 | spec/0011-scalar-indexing.md | Native scalar filtering: B-tree / bitmap algorithms, ScalarIndex Object — draft, Phase 2 |
| 12 | spec/0012-federation.md | Federation Manifest, federate verb (resolves OQ-40), cross-cluster scatter-gather — draft, Phase 3 |
| 13 | spec/0013-graph-indexing.md | Vamana / DiskANN — GraphIndex + GraphPage Objects; router GraphIndex for cross-shard — draft, Phase 3 |
| 14 | spec/0014-streaming-extensions.md | Path A item chunking (ItemManifest), Path B CMAF renditions (RenditionPlaylist) — draft, Phase 3 |
| 15 | spec/0015-hybrid-retrieval.md | TextIndex + MultiVectorIndex; HybridQuery verb; RRF default fusion; query planner cost model — draft, Phase 4 |
| 16 | spec/0016-streaming-freshness.md | HotShard delta tier; dreamdb.fresh-vamana-cosine; index-health signal; bounded-staleness contract — draft, Phase 4 |
| 17 | spec/0017-schema-evolution.md | Multi-version modalities; Reencode verb; compatible_with planner hint — draft, Phase 4 |
| 18 | spec/0018-multi-tenant.md | Per-Space quotas; tenant_id in capability tokens; TenantUsageBatch; fair-share scheduling — draft, Phase 4 |
| 19 | spec/0019-encryption.md | Convergent + per-Space modes; EncryptionMeta header; KMS abstraction — sketch, Phase 4 |
| 20 | spec/0020-tombstones.md | TombstoneListObject; per-Space dreamdb.tombstones registry; read-side suppression filter — Phase 5 |
Open Questions: Audit
87 OQs total across v0 + Phase-3 + Phase-4 drafts. v0 (OQ-1–43): 41 resolved actively, 2 deferred to v0.1+. Phase-3 (OQ-44–61): formally opened by drafts 0010/0012/0013/0014 — most deferred to first implementation pass. Phase-4 (OQ-62–87): formally opened by drafts 0015–0019 — most deferred to first implementation pass; the 9 OQs in 0019 are sketch-level and block the spec/0019 sketch→full-draft promotion.
Resolved within v0 (29 OQs)
| OQ | Topic | Resolution |
|---|---|---|
| OQ-1 | Genesis-relative vs. absolute origin | 0003 §2 — Genesis-relative canonical, optional Unix-ns back-ref |
| OQ-2 | Spatial-indexing scheme | 0004 §5 — dreamdb.lsh-cosine |
| OQ-3 | Backend consistency model | 0005 §5 — strong RAW for content; linearizable CAS for refs |
| OQ-4 | Per-modality container | 0007 §5 — CMAF / fragmented MP4 |
| OQ-5 | Manifest distribution | 0008 §8 — Refs / hash-addressed / pull-only / federation |
| OQ-6 | Modality-tag grammar | 0002 §5 — dot-separated; reverse-DNS for user-defined |
| OQ-7 | Constant correction conflicts | 0008 §6.3 + 0007 §8.2 — lex-greatest layer wins |
| OQ-8 | Event bucketing default | 0007 §8 — Time-batch format; per-modality bucket-duration |
| OQ-9 | Spatial bucket-key derivation | 0004 §5 — ChaCha20-seeded LSH-cosine |
| OQ-10 | Default Fragment duration | 0007 §5.4 — 2 s default; 1–30 s range |
| OQ-11 | Concrete CBOR tag numbers | 0009 §3.2 — private-use range 65521–65525 |
| OQ-12 | Multihash algorithm tags | 0009 §3.1 — 0x1e (IPFS-aligned) |
| OQ-13 | Spatial+time segment order | 0007 §6.5 — spatial-first |
| OQ-14 | Index Page fanout & size | 0007 §7.1 — B = 256, 16 KiB target |
| OQ-15 | Inline-vs-paged threshold | 0007 §7.2 — 1 MiB |
| OQ-16 | Per-modality byte layout | 0007 §6 — fixed-size + offset-table patterns |
| OQ-17 | CBOR tag for time anchor | 0009 §3.2 — 65524 |
| OQ-20 | Index Page delta encoding | 0007 §7.3 — deltas for time; positional arrays |
| OQ-22 | Local-FS Connector | 0005 §9 — HTTP-only in v0; MinIO for dev |
| OQ-23 | Vector storage layout | 0007 §6 — inline (default) vs. reference (multi-table) |
| OQ-24 | Bucket-splitting rule | 0007 §6.4 — 100 MiB threshold |
| OQ-27 | LSH conformance vectors | 0009 §5.3 — multi-arch f32 discipline |
| OQ-28 | Adaptive widening / speculative preload | 0006 §6.4 / §6.5 — implementation-defined |
| OQ-29 | HTTP version requirement | 0005 §6 — HTTP/2 mandatory |
| OQ-30 | HTTP test vectors | 0009 §6 — full battery |
| OQ-33 | Verb conformance vectors | 0009 §7 — full battery |
| OQ-35 | Explicit parent for Publish | 0008 §4.2 — Publish(parents: [...]) |
| OQ-36 | Byte-format vectors | 0009 §5.4 — full battery |
| OQ-39 | DAG conformance vectors | 0009 §8 — full battery |
Decided "no" in v0; may revisit in v0.1+ (2 OQs)
| OQ | Topic | v0 decision |
|---|---|---|
| OQ-19 | Negative time anchors | v0: forbidden (unsigned u64). v0.1 may add bias scheme. |
| OQ-21 | Byte-size delta encoding | v0: as-is, no delta. v0.1 may add if benchmarks justify. |
Deferred to v0.1+ as feature additions (7 OQs)
| OQ | Topic | Rationale |
|---|---|---|
| OQ-18 | Coarser time resolutions (ms, s) | v0 ns is sufficient; revisit if storage waste measured |
| OQ-26 | dreamdb.lsh-l2 algorithm | L2-metric modalities; v0 cosine-only |
| OQ-31 | Multipart upload | v0 caps Bucket Objects at 100 MiB; multipart unneeded |
| OQ-32 | Presigned-URL Connector | Web/browser deployment pattern |
| OQ-34 | Subscribe / Watch verb | v0 is pull-only via Ref polling |
| OQ-37 | Additional media codecs | AV1, HEVC, FLAC, VP9 — all CMAF-compatible |
| OQ-41 | tag namespace | Immutable Refs (release tags) |
Resolved in Phase-3 / Phase-4 spec drafts (4 OQs)
| OQ | Topic | Resolution |
|---|---|---|
| OQ-25 | dreamdb.pq-ivf algorithm | 0010 §7 — superseded by dreamdb.qinco-cosine (NeurIPS 2025); classical PQ sketched in 0010 §11.1 |
| OQ-38 | PQ-compressed vector layout | 0010 §8 — bucket header extends to 200 bytes carrying vector_compressor_hash |
| OQ-40 | federate verb | 0012 §4 — push + pull modes with capability tokens (Ed25519) |
| OQ-54 | Streaming graph updates | 0016 §4 — dreamdb.fresh-vamana-cosine ships append + background consolidation + tombstone deletes |
Deferred to community / governance process (2 OQs)
| OQ | Topic |
|---|---|
| OQ-42 | Public registry of conformance reports |
| OQ-43 | Conformance suite release cadence |
Foundational Doctrines (cross-cutting)
The spec accumulates several disciplines that span multiple documents. Quick reference:
| Doctrine | Defined in | Why |
|---|---|---|
| Time as the sole primary key | 0000 §5.1 | No user-level IDs; everything joins via timeline |
| Strict immutability for content; CAS for refs | 0000 §5.2 | Lock-free data plane; refs are the only mutable thing |
Address = <object-address> · <intra-object-locator> | 0000 §5.3 | Item ≠ Object; bucketing makes 1B-scale viable |
| Streaming-native encapsulation | 0000 §5.4 | Fragment-as-Item; index = seek pointer |
| Manifest Supremacy | 0005 §5.3.1 | Hot path uses cached object_index; LIST is bootstrap-only |
| Universal byte-range URIs | 0002 §6.5 | All dreamdb:// URIs use #bytes:<a>-<b> form; opaque to readers |
| Integer-only time arithmetic | 0003 §4.1 | Zero f64 ops on the time path |
| F32 left-fold determinism | 0004 §5.4 | Bit-identical LSH across hardware paths |
| Pure-scalar reference is canonical | 0009 §5.3.1 | SIMD paths verified against scalar |
| Lineage validation at decode time | 0007 §6.1.1 | Bucket headers carry spatial_index_hash |
| MUST-refuse-on-SpatialIndex-incompat | 0008 §6.5 | Catastrophic silent-merge case at v0; non-conformance gate |
| Feature-branching at scale | 0008 §3.1 | Per-user/team Refs; coordinator merges into refs/main |
| Monotonic ts (observability) | 0008 §7.3 | SHOULD; never trusted for correctness |
What's Next
The v0 spec is complete and the reference implementation is shipping. Current focus:
- Billion-scale validation: imagenet-1k (1.3M records) ingest in progress 2026-05-18; SIFT1M / LAION-100M next.
- Production hardening: observability spec, cross-SDK interop tests, RBAC / capability tokens (deferred to v0.1).
- Tombstone storage compaction (per
spec/0020 §6): operator pass that reclaims bytes for tombstoned anchors. - Multi-parent merge formalization:
design/0007-sharded-ingest.mddocuments the algorithm; may be promoted to its own spec number.
See README.md for the project pitch, status table, and quickstart.
Spec Status
| Field | Value |
|---|---|
| Spec version | v0 + Phase-3 (0010/0012–0014) + Phase-4 (0015–0019) + Phase-5 (0020) |
| Document count | 21 (0000–0020) |
| Open Questions | 87 total — 45 resolved (v0 + Phase-3/4/5); 42 open or deferred to first-implementation pass |
| Reference implementation | shipping (dreamdb-* crates, dreamdb-dataset-python, browser SDK) |
| Conformance suite | partial (CBOR / address / time / spatial covered; cross-SDK interop deferred) |
| Test suite | 721 green across crates |
| Billion-scale benchmark | in progress (1.3M-record imagenet-1k ingest 2026-05-18) |