DreamDB Specification — 0009: Conformance

Status: Draft. Builds on 0000–0008 (the entire v0 spec). This document defines what "DreamDB-conformant" means for each component (Backend, Storage Connector, Protocol Component / SDK), enumerates the test-vector battery that any conformant implementation MUST pass, and resolves the meta-OQs about concrete tag values: OQ-11 (CBOR tag numbers), OQ-12 (multihash algorithm tags), OQ-17 (time-anchor CBOR tag).

1. Purpose

The DreamDB v0 spec defines a protocol composed of byte formats, address grammars, HTTP semantics, algorithms, and operational disciplines. An implementation is DreamDB-conformant if it correctly realizes the relevant subset for its role. This document:

• Defines the three conformance roles (Backend, Storage Connector, Protocol Component) and what each must implement.
• Resolves the spec's meta-OQs — concrete CBOR tag values, multihash algorithm tags, and other registry-style commitments deferred from earlier docs.
• Enumerates the test-vector categories that a conformant implementation MUST pass, organized by source spec doc.
• Specifies the test-vector exchange format — a structured JSON form that is portable across implementation languages.
• Defines the conformance reporting that an implementation publishes to advertise its level.

What this document does not do:

• Enumerate every individual test vector in full. The actual vectors live in an external repository (see §10) maintained alongside the spec; this document defines the categories and pass criteria.
• Replace earlier specs' normative requirements. Conformance is judged against the entire spec; this document is a roadmap, not a substitute.

2. Conformance Roles

DreamDB's three-layer architecture (0000 §3) admits three conformance roles, each independently testable:

A complete implementation is two of these (Connector + Protocol; the Backend is supplied by an existing object store) plus a binding to a chosen Backend that has independently been verified Backend-conformant.

2.1 Backend conformance

A Backend is ContentStore-Conformant if it correctly implements:

• HTTP verbs PUT, GET (with Range), HEAD, LIST (with prefix and pagination), DELETE per 0005 §3.
• Strong read-after-write consistency for content-addressed paths (0005 §5.1).
• Lexicographic byte-order LIST results (0005 §3.5.1).
• HTTP/2 (or HTTP/3) connectivity (0005 §6).

A Backend is RefStore-Conformant if, additionally, it correctly implements:

• If-None-Match: * and If-Match: <etag> conditional writes per 0005 §4.
• Linearizable CAS semantics (0005 §5.2).

Each tier is independently certifiable; a Backend MAY be ContentStore-Conformant only.

2.2 Storage Connector conformance

A Storage Connector is conformant if it:

• Translates DreamDB protocol-layer requests into HTTP per 0005.
• Iterates LIST pagination to exhaustion (0005 §3.5.2).
• Round-trips ETags opaquely for If-Match and canonicalizes for cross-request comparison (0005 §8.5).
• Issues HTTP/2 multiplexed streams over a shared connection per (backend host, auth identity) pair (0005 §8.1).
• Translates bytes:<a>-<b> (DreamDB half-open) → Range: bytes=<a>-<b−1> (HTTP inclusive) (0005 §3.3).
• Exposes no DreamDB-semantic logic — bytes pass through opaquely (0005 §8.4).

2.3 Protocol Component (SDK) conformance

A Protocol Component is conformant if it correctly implements:

• The eight verbs of 0006 §2 (Open, Resolve, Query, Stream, Get, Append, Layer, Publish).
• The per-session cache discipline of 0006 §3 (cache by content hash; never by path).
• Manifest Supremacy — never use list-prefix during steady-state reads (0005 §5.3.1).
• The integer-only time arithmetic discipline of 0003 §4.1.
• The f32-determinism discipline for spatial-key derivation (0004 §5.4).
• The deterministic CBOR encoding of all hashable Objects (0002 §3).
• The base2 spatial-key encoding, base32 hash encoding, 16-char hex time encoding of 0002 §6 / §8 and 0003 §6.
• The byte-format layouts of 0007 for every Object kind it produces or reads.
• The DAG semantics of 0008 — multi-parent Manifests, MUST-refuse-on-SpatialIndex-conflict merges, lex-greatest tiebreak for Constants.
• The operational disciplines of 0006 §4.1.1, §4.4.1, §7.3 (Ref freshness, Stream prefetch, GC).

3. Resolved Meta-OQs

This section resolves the placeholder values used in earlier docs.

3.1 Multihash algorithm tag for BLAKE3 (resolves OQ-12)

BLAKE3-256 algorithm tag = 0x1e

DreamDB aligns with the IPFS multihash table for BLAKE3 (0x1e). This commits DreamDB's algorithm-tag namespace to IPFS-compatible values for any future hash function as well — implementations MAY consult the IPFS multihash registry when adding hash algorithms in v0.1+.

A multihash for BLAKE3-256 on the wire is therefore:

0x1e || <32-byte BLAKE3 hash>      = 33 bytes total

Encoded for use in addresses as base32-lowercase-no-padding (per 0002 §8.1): 53 characters (33 bytes × 8 bits / 5 bits per char = ceil(52.8) = 53).

3.2 DreamDB CBOR tag value (resolves OQ-11, OQ-17)

DreamDB reserves one CBOR tag in the private-use range for foreign-CBOR-embedding scenarios:

dreamdb.tag = 65521 (= 0xFFF1)

Selected for v0 because:

• Already designated for private use by RFC 8949 §6.
• No conflict with IANA-registered CBOR tags.
• Implementations don't need to coordinate with a registry; the tag is DreamDB-private.
• v0.1 MAY apply for IANA registration in the public-use range and renumber, with backward-compatibility readers.

3.2.1 Tag scope: foreign-CBOR contexts only

The tag is explicitly reserved for unstructured / schema-less CBOR contexts — situations where a DreamDB value (a hash, a time anchor, a modality tag) is embedded in a generic CBOR document outside the spec'd DreamDB schemas, and a reader needs the tag to disambiguate it from a generic integer / byte string / text string.

Inside the schema-typed fields of DreamDB Objects, this tag is NOT used. Those fields use plain CBOR types directly:

• Time anchors → plain CBOR uint (per 0003 §5).
• Hashes → plain CBOR byte string (per 0002 §3).
• Modality tags → plain CBOR text string.
• Spatial keys → plain CBOR text string (base2 chars).

The schema's field name (or array position) is the type indicator. Adding a tag inside DreamDB schemas would be redundant and adds 2–3 bytes per occurrence — at billion-scale, that is GB-class waste.

If your application doesn't embed DreamDB values in foreign CBOR documents, you never encounter dreamdb.tag. The v0 reference implementation does not produce or consume it on any path.

3.2.2 Maps vs. positional arrays in DreamDB schemas

0002 §3.1 specifies two encoding choices for schema-typed DreamDB Objects:

• Maps with string keys: low-volume schemas (Manifest top-level, Genesis, SpatialIndex Object, Track Object metadata). Self-describing; debugger-friendly.
• Positional CBOR arrays: high-volume schemas (Index Page leaf entries and internal entries per 0007 §7.3 / §7.5). Field order pinned in spec text; saves ~40 bytes per entry and ~40 GB on a 1B-entry Track.

Conformance test vectors (§5.1) MUST cover both encodings — including a positional-array round-trip that confirms the field order matches the spec.

4. Test-Vector Format

Conformance test vectors are exchanged as JSON documents with the following schema:

{
   "test_id":      "0002.cbor.deterministic.0001",
   "spec_section": "0002 §3.1",
   "category":     "cbor-roundtrip",
   "description":  "Round-trip a Manifest with one Track entry",
   "inputs":       { ... category-specific inputs ... },
   "expected":     { ... category-specific expected outputs ... },
   "pass_criteria": "byte-identical round-trip",
   "metadata":     { ... optional, e.g. golden-file references ... }
}

4.1 Test ID convention

test_id := <spec-doc>.<category>.<subcategory>.<sequence>

Example IDs:

• 0002.cbor.deterministic.0001 — first deterministic-CBOR test in 0002.
• 0004.lsh-cosine.f32-rounding.0003 — third f32-rounding-edge-case test for dreamdb.lsh-cosine.
• 0005.http.range-translation.0001 — first byte-range translation test.
• 0008.dag.refuse-spatial-index-incompat.0001 — first MUST-refuse merge test.

This makes failures locatable and grep-friendly.

4.2 Categories

Each category specifies its own input/expected schema. The full schema definitions live in the conformance suite repo (§10); summary follows in §5–§8.

4.3 Pass / fail criteria

Vectors specify their own pass criteria. Common kinds:

• Byte-identical: produced output bytes equal expected bytes exactly.
• Functional equivalence: produced output is semantically equivalent (e.g. an array order-independent set comparison).
• Behavioral assertion: under specified conditions, the implementation makes (or does not make) certain HTTP requests; takes (or does not take) certain actions.
• Tolerance bound: produced metric (e.g. recall, latency) is within a stated bound of the expected value.

5. Test Categories — Byte Formats and Algorithms

5.1 0002: Content addressing

5.2 0003: Time encoding

5.3 0004: Spatial indexing

5.3.1 Multi-architecture f32-determinism conformance (mandatory)

Floating-point determinism across heterogeneous hardware is the single most fragile conformance surface. SIMD optimizers (AVX-512 fused-multiply-add, ARM NEON's vector-friendly rounding modes, GPU paths) are at liberty to reorder f32 operations; the resulting drift is invisible until two SDKs disagree on a single bit and silently route data to different Spatial Buckets.

The discipline:

Pure-scalar reference is the canonical implementation. The dreamdb.lsh-cosine algorithm spec (0004 §5) is interpreted to mean the bytes the scalar reference path produces, with all SIMD optimizations disabled. Any SIMD path is conformant if and only if it produces bit-identical output to scalar.

Required hardware coverage: a conformant implementation MUST be tested on the following paths, all producing bit-identical results to the scalar reference:

Test-vector design: vectors crafted to expose rounding-order sensitivity:

• Pairs with dot products at exactly 0.5 ulp (the rounding boundary).
• Vectors where left-fold and right-fold accumulation order produce different f32 results — and where the spec mandates left-fold (0004 §5.4).
• Adversarial inputs: extreme magnitude ranges, near-zero dot products, denormalized floats.
• Pairs at exactly the <v, h_i> = 0 tie boundary (0004 §5.1 specifies tie → 1).

Non-conformance: an implementation that produces different bits under SIMD vs. scalar on any required hardware path is non-conformant for that path. The implementation MUST disable SIMD optimization on that platform until parity is achieved. Compromised SIMD paths shipping silently is the failure mode this discipline exists to prevent.

This is the same model used by cryptographic library testing (BoringSSL, RustCrypto, libsodium): bit-identical scalar reference, hardware paths verified against it, no SIMD path ships without bit parity.

5.4 0007: Object byte formats

6. Test Categories — HTTP Contract (0005)

6.1 ContentStore HTTP semantics

6.2 RefStore HTTP semantics

6.3 Connector behaviors

7. Test Categories — Verbs and SDK behavior (0006)

8. Test Categories — DAG and Versioning (0008)

8.5 Test Categories — Phase-3 ObjectKinds and Verbs

Phase-3 spec drafts (0010, 0012, 0013, 0014) introduce new ObjectKinds and one new verb. The conformance battery for them is OPTIONAL for pre-Phase-3 v0 SDKs and REQUIRED for SDKs claiming Phase-3 conformance. Specific vectors are deferred to a v0.X amendment of this spec but the categories are pinned now so implementations have known targets:

8.5.1 0010 — Vector compression

8.5.2 0012 — Federation

8.5.3 0013 — Graph indexing

8.5.4 0014 — Streaming extensions

8.5.5 New ObjectKind path-grammar coverage

8.6 Test Categories — Phase-4 ObjectKinds and Verbs

Phase-4 spec drafts (0015, 0016, 0017, 0018, 0019) introduce hybrid retrieval, streaming freshness, schema evolution, multi-tenancy, and encryption. Conformance for them is OPTIONAL for pre-Phase-4 SDKs and REQUIRED for SDKs claiming Phase-4 conformance. Specific vectors are deferred to a v0.X amendment of this spec; categories are pinned here as known targets.

8.6.1 0015 — Hybrid retrieval

8.6.2 0016 — Streaming freshness

8.6.3 0017 — Schema evolution

8.6.4 0018 — Multi-tenant

8.6.5 0019 — Data-plane encryption (sketch-level)

Conformance for encryption is provisional pending the v0.X+1 promotion of spec/0019 from sketch to full draft. Provisional categories:

8.6.6 Phase-4 path-grammar coverage

9. Test Categories — End-to-End Worked Example

A full end-to-end test that exercises the entire spec in one scenario:

test_id: e2e.video-search.0001
description: Ingest a 10-minute video; embed; query "moment of interest"; stream.

Setup:
  - Backend: MinIO localhost:9000.
  - Storage Connector: HTTP/2 to localhost.
  - SDK: any conformant Protocol Component.

Steps:
  1. Open(backend, "main") — should be cold; LIST manifests/ to bootstrap.
  2. Append(video.h264, [600 fragments × 1s each]) — single-shot or paged.
  3. Layer(embedding.f32.dim=768.bucketed.spatial-bits=18, derived from video).
  4. Append(title.text, ["Test Recording"]).
  5. Publish — advance refs/main via CAS.
  6. (Disconnect the SDK, wait 5 s, reopen — verify Ref freshness.)
  7. Open(backend, "main") — fetches the new Manifest.
  8. Query(embedding modality, query_vector, k=5, recall=0.9) — measure latency.
  9. Stream(video, time_range around top result, +/- 5 s) — verify decoder accepts.
  10. (Run GC with 24h threshold; verify nothing reachable was deleted.)

Pass criteria:
  - All HTTP responses match expected status codes.
  - Manifest published is reachable from refs/main.
  - Cold-start query latency < 200 ms p95.
  - Hot-path subsequent query (same SDK session) < 100 ms p95.
  - Stream first-byte < 100 ms.
  - GC preserves all Objects reachable from the test's Manifest.
  - Throughout: ZERO LIST HTTP requests on the hot path.

10. Conformance Suite Repository

The actual test vectors and runner code live in an external repository (designed to live at dreamdb-conformance adjacent to the spec repo):

dreamdb-conformance/
├── README.md
├── runners/
│   ├── connector-runner.rs        — exercises a Storage Connector against MinIO
│   ├── protocol-runner.rs         — exercises an SDK end-to-end
│   └── backend-runner.rs          — verifies HTTP-level backend conformance
├── vectors/
│   ├── 0002/
│   │   ├── cbor.deterministic/
│   │   │   ├── 0001.json
│   │   │   ├── 0001.expected.cbor
│   │   │   └── ...
│   │   └── ...
│   ├── 0003/
│   ├── 0004/
│   └── ...
└── reports/
    └── ...                         — per-implementation conformance reports

Vectors are JSON per §4 with binary expected-output files alongside (.expected.cbor, .expected.bin). The runners are reference implementations in Rust; ports to other languages are welcome.

A v0 implementation publishes a conformance report (a structured JSON document listing which test categories passed/failed) and a chosen Tier ("ContentStore-Conformant Backend," "Full Protocol Component," etc.) when claiming DreamDB compatibility.

11. Self-Certification

Until a formal conformance authority exists, implementations self-certify:

1. Run the conformance suite against your implementation.
2. Publish the resulting report (machine-readable JSON + human-readable summary).
3. State your claimed Tier in the README.
4. Cite the spec version (e.g. "DreamDB v0.1.0").

The community SHOULD maintain a public list of self-certified implementations and their reports. Discrepancies between claimed and actual conformance can be challenged via the public report; this is the same model used by Web Platform Tests, IETF interop reports, and similar.

A formal certification authority MAY emerge in v0.1+; v0 leans on community-maintained transparency.

12. Compatibility Across Spec Revisions

DreamDB spec versions follow semantic versioning:

• Patch (v0.1.x → v0.1.y): clarifications, OQ resolutions, conformance test additions. Existing implementations remain conformant.
• Minor (v0.1 → v0.2): feature additions (new modalities, new algorithms, new optional verbs). Existing implementations remain conformant for the subset they support; new features are opt-in.
• Major (v0 → v1): breaking changes. Implementations MAY support multiple major versions in parallel via a version-prefixed namespace (TBD in v1).

Within v0:

• v0 freezes the data model, address grammar, time encoding, hash function, and HTTP contract.
• v0.1+ MAY add: new spatial-indexing algorithms (dreamdb.pq-ivf, dreamdb.lsh-l2, dreamdb.learned-mlp-v1), additional modalities (AV1, HEVC, FLAC, sensor types), federation primitives, named tags, push-style notifications.

13. Open Questions Resolved by This Document

This document explicitly resolves:

• OQ-11 (concrete CBOR tag numbers): § 3.2.
• OQ-12 (multihash algorithm tags — IPFS-aligned): § 3.1.
• OQ-17 (concrete value for the time-anchor CBOR tag): § 3.2 — folded into the single dreamdb.tag = 65521 reserved for foreign-CBOR contexts; not used inside DreamDB schemas.

All other "OQ-NN → 0009" pointers from 0000–0008 are resolved by the test-vector categories enumerated in §5–§9.

14. Open Questions Not Resolved

Two operational decisions deferred to community / operator coordination:

• OQ-42 (→ community / future spec): Public registry of conformance reports. v0 is self-certification only; a formal authority may emerge.
• OQ-43 (→ community / future spec): Publication cadence for the conformance suite repository. v0 is unscheduled — releases tied to spec revisions.

End of v0 specification. Implementations interested in v0 conformance should start with 0000-overview.md, build forward through 0001–0008, and verify against the test categories in this document.