DreamDB Specification — 0009: Conformance

Status: Draft. Builds on 0000–0008 (the entire v0 spec). This document defines what "DreamDB-conformant" means for each component (Backend, Storage Connector, Protocol Component / SDK), enumerates the test-vector battery that any conformant implementation MUST pass, and resolves the meta-OQs about concrete tag values: OQ-11 (CBOR tag numbers), OQ-12 (multihash algorithm tags), OQ-17 (time-anchor CBOR tag).

1. Purpose

The DreamDB v0 spec defines a protocol composed of byte formats, address grammars, HTTP semantics, algorithms, and operational disciplines. An implementation is DreamDB-conformant if it correctly realizes the relevant subset for its role. This document:

• Defines the three conformance roles (Backend, Storage Connector, Protocol Component) and what each must implement.
• Resolves the spec's meta-OQs — concrete CBOR tag values, multihash algorithm tags, and other registry-style commitments deferred from earlier docs.
• Enumerates the test-vector categories that a conformant implementation MUST pass, organized by source spec doc.
• Specifies the test-vector exchange format — a structured JSON form that is portable across implementation languages.
• Defines the conformance reporting that an implementation publishes to advertise its level.

What this document does not do:

• Enumerate every individual vector in full. The reference corpus lives in dreamdb-conformance/vectors/ in this repository; §10 describes its exchange organization. Category rows define requirements or named performance profiles, not proof that every row has an implemented test or has run on every platform.
• Replace earlier specs' normative requirements. Conformance is judged against the entire spec; this document is a roadmap, not a substitute.

2. Conformance Roles

DreamDB's three-layer architecture (0000 §3) admits three conformance roles, each independently testable:

A complete implementation is two of these (Connector + Protocol; the Backend is supplied by an existing object store) plus a binding to a chosen Backend that has independently been verified Backend-conformant.

2.1 Backend conformance

A Backend is ContentStore-Conformant if it correctly implements:

• HTTP verbs PUT, GET (with Range), HEAD, LIST (with prefix and pagination), DELETE per 0005 §3.
• Strong read-after-write consistency for content-addressed paths (0005 §5.1).
• Lexicographic byte-order LIST results (0005 §3.5.1).
• The §3–§5 HTTP semantics over HTTP/1.1 or later (0005 §6). HTTP/2 / HTTP/3 is a performance upgrade (a profile attribute, 0023), not a conformance requirement — the byte contract is identical across versions.

A Backend is RefStore-Conformant if, additionally, it correctly implements:

• If-None-Match: * and If-Match: <etag> conditional writes per 0005 §4.
• Linearizable CAS semantics (0005 §5.2).

Each tier is independently certifiable; a Backend MAY be ContentStore-Conformant only.

2.2 Storage Connector conformance

A Storage Connector is conformant if it:

• Translates DreamDB protocol-layer requests into HTTP per 0005.
• Iterates LIST pagination to exhaustion (0005 §3.5.2).
• Round-trips ETags opaquely for If-Match and canonicalizes for cross-request comparison (0005 §8.5).
• Reuses connections per (backend host, auth identity) pair rather than one-per-request (0005 §8.1); when it negotiates HTTP/2 / HTTP/3 those requests share the connection's multiplexed streams. The multiplexing is a performance-profile attribute (0023), not a conformance gate; the connector is conformant over HTTP/1.1.
• Translates bytes:<a>-<b> (DreamDB half-open) → Range: bytes=<a>-<b−1> (HTTP inclusive) (0005 §3.3).
• Exposes no DreamDB-semantic logic — bytes pass through opaquely (0005 §8.4).

2.3 Protocol Component (SDK) conformance

A Protocol Component is conformant if it correctly implements:

• The eight verbs of 0006 §2 (Open, Resolve, Query, Stream, Get, Append, Layer, Publish).
• The per-session cache discipline of 0006 §3 (cache by content hash; never by path).
• Manifest Supremacy — never use list-prefix during steady-state reads (0005 §5.3.1).
• The integer-only time arithmetic discipline of 0003 §4.1.
• The f32-determinism discipline for spatial-key derivation (0004 §5.4).
• The deterministic CBOR encoding of all hashable Objects (0002 §3).
• The base2 spatial-key encoding, base32 hash encoding, 16-char hex time encoding of 0002 §6 / §8 and 0003 §6.
• The byte-format layouts of 0007 for every Object kind it produces or reads.
• The DAG semantics of 0008 — multi-parent Manifests, MUST-refuse-on-SpatialIndex-conflict merges, lex-greatest tiebreak for Constants.
• The operational disciplines of 0006 §4.1.1, §4.4.1, §7.3 (Ref freshness, Stream prefetch, GC).

3. Resolved Meta-OQs

This section resolves the placeholder values used in earlier docs.

3.1 Multihash algorithm tag for BLAKE3 (resolves OQ-12)

BLAKE3-256 algorithm tag = 0x1e

DreamDB uses the BLAKE3 algorithm code 0x1e, but its fixed 33-byte wire value below is DreamDB's encoding, not a complete standard multihash serialization: there is no digest-length varint. Matching an algorithm code does not make the encodings interchangeable. A future algorithm requires an explicit protocol revision; readers must not enable it merely because an external registry lists it.

A multihash for BLAKE3-256 on the wire is therefore:

0x1e || <32-byte BLAKE3 hash>      = 33 bytes total

Encoded for use in addresses as base32-lowercase-no-padding (per 0002 §8.1): 53 characters (33 bytes × 8 bits / 5 bits per char = ceil(52.8) = 53).

3.2 DreamDB CBOR tag value (resolves OQ-11, OQ-17)

DreamDB specifies one locally chosen CBOR tag for foreign-CBOR-embedding scenarios:

dreamdb.tag = 65521 (= 0xFFF1)

This is not an IANA private-use allocation. The IANA CBOR Tags registry places 32768 and above under First Come First Served registration, not a private-use policy. This document does not establish a DreamDB registration or global collision freedom. Foreign-CBOR integrations must agree on the local meaning; any registration or renumbering needs a separate compatibility decision. This correction does not change the value or introduce tags into DreamDB Object schemas (§3.2.1).

3.2.1 Tag scope: foreign-CBOR contexts only

The tag is a deployment-local foreign-CBOR convention. An integration MUST agree both its use of 65521 and the wrapped semantic type out of band. Seeing 65521 in an arbitrary document is not sufficient evidence of DreamDB content; without that agreement a consumer preserves it as an uninterpreted tag or refuses the unsupported application format, rather than interpreting it as a DreamDB value. The wrapped CBOR major type alone is insufficient (modality tags and spatial keys are both strings).

Inside the schema-typed fields of DreamDB Objects, this tag is NOT used. Those fields use plain CBOR types directly:

• Time anchors → plain CBOR uint (per 0003 §5).
• Hashes → plain CBOR byte string (per 0002 §3).
• Modality tags → plain CBOR text string.
• Spatial keys → plain CBOR text string (base2 chars).

The schema's field name (or array position) is the type indicator. Adding a tag inside DreamDB schemas would be redundant and adds 2–3 bytes per occurrence — at billion-scale, that is GB-class waste.

Ordinary DreamDB Object readers do not acquire tagged alternatives to their fields. The reference core exports DREAMDB_TAG and its generic CBOR codec can round-trip a tagged value, but this does not perform automatic DreamDB semantic interpretation or establish an interoperable foreign-document format.

3.2.2 Maps vs. positional arrays in DreamDB schemas

0002 §3.1 specifies two encoding choices for schema-typed DreamDB Objects:

• Maps with string keys: low-volume schemas (Manifest top-level, Genesis, SpatialIndex Object, Track Object metadata). Self-describing; debugger-friendly.
• Positional CBOR arrays: high-volume schemas (Index Page leaf entries and internal entries per 0007 §7.3 / §7.5). Field order pinned in spec text; saves ~40 bytes per entry and ~40 GB on a 1B-entry Track.

Conformance test vectors (§5.1) MUST cover both encodings — including a positional-array round-trip that confirms the field order matches the spec.

4. Test-Vector Format

Conformance test vectors are exchanged as JSON documents with the following schema:

{
   "test_id":      "0002.cbor.deterministic.0001",
   "spec_section": "0002 §3.1",
   "category":     "cbor-roundtrip",
   "description":  "Round-trip a Manifest with one Track entry",
   "inputs":       { ... category-specific inputs ... },
   "expected":     { ... category-specific expected outputs ... },
   "pass_criteria": "byte-identical round-trip",
   "metadata":     { ... optional, e.g. golden-file references ... }
}

4.1 Test ID convention

test_id := <spec-doc>.<category>.<subcategory>.<sequence>

Example IDs:

• 0002.cbor.deterministic.0001 — first deterministic-CBOR test in 0002.
• 0004.lsh-cosine.f32-rounding.0003 — third f32-rounding-edge-case test for dreamdb.lsh-cosine.
• 0005.http.range-translation.0001 — first byte-range translation test.
• 0008.dag.refuse-spatial-index-incompat.0001 — first MUST-refuse merge test.

This makes failures locatable and grep-friendly.

4.2 Categories

Each category specifies its own input/expected schema. The full schema definitions live in the conformance suite repo (§10); summary follows in §5–§8.

4.3 Pass / fail criteria

Vectors specify their own pass criteria. Common kinds:

• Byte-identical: produced output bytes equal expected bytes exactly.
• Functional equivalence: produced output is semantically equivalent (e.g. an array order-independent set comparison).
• Behavioral assertion: under specified conditions, the implementation makes (or does not make) certain HTTP requests; takes (or does not take) certain actions.
• Tolerance bound: produced metric (e.g. recall, latency) is within a stated bound of the expected value.

4.4 Conformance vs. performance (normative)

The four kinds in §4.3 fall into two classes with different normative force, and an implementation MUST be evaluated against them separately:

• Conformance criteria — the byte-identical, functional-equivalence, and behavioral-assertion kinds. These are deterministic and environment-independent: given the same inputs, a correct implementation passes on any hardware, network, or workload. An implementation is DreamDB-conformant for a category if and only if it passes every conformance criterion in that category. A failure here is non-conformance.
• Performance-profile targets — the tolerance-bound kind (recall, latency, throughput, bytes-fetched, wall-clock). These depend on the workload (corpus size, dimensionality, distribution) and the environment (storage round-trip, bandwidth, CPU/GPU). They are advisory: missing a target means the implementation does not claim that performance profile — it does not make the implementation non-conformant. Performance profiles are defined separately in 0023-performance-profiles.md and version on their own cadence.

The boundary must be read carefully: a quantity fixed by the protocol for the declared fixture is a behavioral assertion, not a timing target. For example, Manifest Supremacy requires zero query LISTs. An exact Bucket GET count additionally depends on the fixture's split count, cache state, coalescing and retry conditions; it cannot be inferred from the number of tables alone. “Completes in <100 ms p95” is a performance-profile target.

Performance narrative elsewhere in the spec (e.g. 0004 §7.3/§8 latency budgets) is informative. The normative requirements such narrative rests on — chiefly Manifest Supremacy (0005 §5.3.1: no list-prefix on the steady-state read path) and the cold-start round-trip bound — are stated as conformance criteria in their own right (§6, §7), never inferred from a budget.

5. Test Categories — Byte Formats and Algorithms

5.1 0002: Content addressing

multihash.encoding.* covers whole-Object hashing. bao.range-integrity.* separately pins the derived proof bytes and the verified-range algorithm from 0002 §6.5.4; passing one category does not imply passing the other.

5.2 0003: Time encoding

5.3 0004: Spatial indexing

0004.lsh_l2.001–002 pin the optional dreamdb.lsh-l2 profile (§5.8 of 0004): canonical SpatialIndex hash, seeded signed-bin/probe order, raw negative squared L2 score, and decoder rejection of canonical CBOR containing a zero bucket width. Category lsh-l2-determinism expresses the seed and binary64 width as little-endian hex bytes; query/candidate JSON numbers become binary32 inputs. This is deterministic wire/arithmetic evidence, not a measurement of global ANN recall or independent-implementation agreement.

5.3.1 Multi-architecture f32-determinism conformance (mandatory)

Floating-point determinism across heterogeneous hardware is the single most fragile conformance surface. SIMD optimizers (AVX-512 fused-multiply-add, ARM NEON's vector-friendly rounding modes, GPU paths) are at liberty to reorder f32 operations; the resulting drift is invisible until two SDKs disagree on a single bit and silently route data to different Spatial Buckets.

The discipline:

Pure-scalar reference is the canonical implementation. The dreamdb.lsh-cosine algorithm spec (0004 §5) is interpreted to mean the bytes the scalar reference path produces, with all SIMD optimizations disabled. Any SIMD path is conformant if and only if it produces bit-identical output to scalar.

Required hardware coverage: a conformant implementation MUST be tested on the following paths, all producing bit-identical results to the scalar reference:

Test-vector design: vectors crafted to expose rounding-order sensitivity:

• Pairs with dot products at exactly 0.5 ulp (the rounding boundary).
• Vectors where left-fold and right-fold accumulation order produce different f32 results — and where the spec mandates left-fold (0004 §5.4).
• Adversarial inputs: extreme magnitude ranges, near-zero dot products, denormalized floats.
• Pairs at exactly the <v, h_i> = 0 tie boundary (0004 §5.1 specifies tie → 1).

Non-conformance: an implementation that produces different bits under SIMD vs. scalar on any required hardware path is non-conformant for that path. The implementation MUST disable SIMD optimization on that platform until parity is achieved. Compromised SIMD paths shipping silently is the failure mode this discipline exists to prevent.

This is the same model used by cryptographic library testing (BoringSSL, RustCrypto, libsodium): bit-identical scalar reference, hardware paths verified against it, no SIMD path ships without bit parity.

5.4 0007: Object byte formats

6. Test Categories — HTTP Contract (0005)

6.1 ContentStore HTTP semantics

6.2 RefStore HTTP semantics

6.3 Connector behaviors

Profile-only (not a conformance gate): connector.h2.shared-connection.* — multiple parallel requests share one HTTP/2 (or HTTP/3) multiplexed connection. This is a 0023 performance-profile attribute (0023 §"environment_class"), exercised only when the profile asserts an HTTP/2+ environment; it is not required for connector conformance (§2.2, 0005 §6). An HTTP/1.1 connector that reuses connections via keep-alive (connector.connection-reuse.* above) is fully conformant.

7. Test Categories — Verbs and SDK behavior (0006)

8. Test Categories — DAG and Versioning (0008)

8.5 Test Categories — Phase-3 ObjectKinds and Verbs

Phase-3 specs (0010, 0012, 0013, 0014) introduce new ObjectKinds and query/write extensions. The conformance battery for them is OPTIONAL for pre-Phase-3 v0 SDKs and REQUIRED for SDKs claiming Phase-3 conformance.

8.5.1 0010 — Vector compression

v0.1 normative for the framework + raw-f32 / rabitq-cosine / pq-cosine. The Object byte format and RaBitQ encode-determinism are vector-gated (dreamdb-conformance/vectors/0010/); the rerank-recall target is a 0023 performance profile, not a conformance gate.

8.5.2 0012 — Federation

8.5.3 0013 — Graph indexing

8.5.4 0014 — Streaming extensions

8.5.5 New ObjectKind path-grammar coverage

8.6 Test Categories — Phase-4 ObjectKinds and Verbs

Phase-4 specifications (0015, 0016, 0017, 0018, 0019) introduce hybrid retrieval, streaming freshness, schema evolution, multi-tenancy, and encryption. Conformance for them is OPTIONAL for pre-Phase-4 SDKs and REQUIRED for SDKs claiming the corresponding Phase-4 capability. Categories without shipped vectors remain known targets; 0019 now has mandatory format vectors in §8.6.5.

8.6.1 0015 — Hybrid retrieval

8.6.2 0016 — Streaming freshness

8.6.3 0017 — Schema evolution

8.6.4 0018 — Multi-tenant

8.6.5 0019 — Data-plane encryption

0019 is a normative wire format, but the reference Dataset/SDK does not yet implement it. The vectors in dreamdb-conformance/vectors/0019/ are mandatory for any implementation claiming Lineage-v3 encryption capability. Passing them establishes the deterministic format and cryptographic decisions below; it does not certify a cloud KMS adapter, key-destruction procedure or plaintext-handling environment.

The vector runner supplies a raw DWK as the result of a successful fixture KMS slot. That is an input to the protocol algorithm, not a substitute test for KMS authentication. KMS call rate, key-cache hit ratio and throughput belong to implementation operations/performance evidence, not conformance.

8.6.6 Phase-4 path-grammar coverage

8.7 0021 — Compaction

Categories for the bucket/tombstone compaction pass (0021). Several overlap the §8 merge categories — compaction must preserve query results exactly (0008 §5.3 read-equivalence).

8.8 0022 — Fragment packs

Categories for many-Items-per-Object packing (0022); the byte-range reader obligation is 0002 §6.5.4.

8.9 0020 — Tombstones (read-side deletion)

Categories for the legacy TombstoneListObject, the persistent paged TombstoneIndexObject, and the read-side suppression filter (0020). The byte-format categories are vector-gated (dreamdb-conformance/vectors/0020/); the suppression/time-travel behavior is a behavioral assertion. Physical reclamation is delegated to 0021 (see §8.7) per 0020 §6.

8.10 0011 — Scalar indexing

Categories for native scalar filtering (0011). v0.2 retains dreamdb.bitmap-categorical for categorical / bool equality and adds byte-pinned value-keyed B-trees for integer/timestamp, finite-float and UTF-8 lexical ordering.

8.11 0024 — Embedding spec identity

Vectors in the embedding-spec-identity category spell realization_commitment, parity.against and artifact digests in the canonical base32 string form of 0002 §8.1, and parity.mean_f64le / parity.min_f64le as 16 lowercase hex characters — the 8 bytes of the IEEE-754 binary64 value, little-endian. A runner decodes both vector spellings locally, because the shared JSON-to-CBOR vector helper has no byte-string convention and teaching it one would change how every other category decodes. The hash-bearing digest fields are promoted to byte strings before canonical encoding; the statistics are decoded straight to binary64 for the threshold comparison and are never rebuilt as CBOR here. No CBOR float appears anywhere in the category, in keeping with 0002 §3.1.5.

Every vector nests the CLOSED embed_spec of 0024 §3 — identity_basis, realization_manifest, implementation_records — under an embed_spec key, with fixture metadata (test_id, category, modality, the expectations) outside it. The nesting is what makes the CLOSED check meaningful: at the document root an unknown fourth wire key cannot be told apart from a new metadata key, so a runner reading the three fields from the root would declare the container closed while accepting anything.

implementation_records is always an array, never a singleton shorthand, never empty, and sorted ascending by each record's canonical CBOR bytes with byte-identical entries refused. Computing that order is the one place in the corpus where a record is encoded as the Manifest would carry it — base32 digests and the two hex statistics resolved to their byte strings first.

A runner recomputes the threshold verdict from the stated mean_f64le, min_f64le and count rather than trusting a verdict stated in the record, and checks the structural rules of 0024 §3.2 — including that the vector's realization_manifest hashes to the realization_commitment its identity_basis declares. It does not execute an encoder and therefore does not compute parity itself; the measurement of 0024 §4 is out of scope for the vector corpus.

8.12 0025 — Typed Array Items

The post-v1 vectors are executable format references, not evidence that a Dataset or SDK writer exists. Dense-v1 vectors and bytes are unchanged.

8.13 0027 — Progressive geometry

geometry-item vectors supply canonical descriptor bytes as cbor_hex and require either byte-identical decode/re-encode with expected_family, or the specific expected_error. They invoke the actual CLOSED protocol decoder; they do not establish Dataset publication, range authentication or GC behavior. Those claims are exercised separately at the public Dataset and SDK boundary.

9. Test Categories — End-to-End Worked Example

A full end-to-end test that exercises the entire spec in one scenario:

test_id: e2e.video-search.0001
description: Ingest a 10-minute video; embed; query "moment of interest"; stream.

Setup:
  - Backend: MinIO localhost:9000.
  - Storage Connector: HTTP/1.1 or later (the HTTP/2 variant exercises the `0023` performance profile, not conformance).
  - SDK: any conformant Protocol Component.

Steps:
  1. Open(backend, "main") — should be cold; LIST manifests/ to bootstrap.
  2. Append(video.h264, [600 fragments × 1s each]) — single-shot or paged.
  3. Layer(embedding.f32.dim=768.bucketed.spatial_bits=18, derived from video).
  4. Append(title.text, ["Test Recording"]).
  5. Publish — advance refs/main via CAS.
  6. (Disconnect the SDK, wait 5 s, reopen — verify Ref freshness.)
  7. Open(backend, "main") — fetches the new Manifest.
  8. Query(embedding modality, query_vector, k=5, recall=0.9) — measure latency.
  9. Stream(video, time_range around top result, +/- 5 s) — verify decoder accepts.
  10. (Run GC with 24h threshold; verify nothing reachable was deleted.)

Conformance criteria (normative — pass/fail, environment-independent):
  - All HTTP responses match expected status codes.
  - Manifest published is reachable from refs/main.
  - GC preserves all Objects reachable from the test's Manifest.
  - Throughout: ZERO LIST HTTP requests on the hot path (Manifest Supremacy).

Performance-profile targets (informative — advisory, NOT pass/fail; see 0023):
  - Cold-start query latency < 200 ms p95.
  - Hot-path subsequent query (same SDK session) < 100 ms p95.
  - Stream first-byte < 100 ms.

10. Conformance Suite Repository

The actual reference corpus and runner live in this repository:

dreamdb-conformance/
├── src/                — reference vector dispatch and checks
├── tests/              — behavioral and corpus integration tests
└── vectors/            — spec-numbered portable inputs and expected outputs

Vectors use the JSON conventions in §4 and category-specific sections, with binary companion files where needed. The Rust runner is one implementation; sharing it across language bindings does not constitute an independent port.

A v0 implementation publishes a conformance report (a structured JSON document listing which test categories passed/failed) and a chosen Tier ("ContentStore-Conformant Backend," "Full Protocol Component," etc.) when claiming DreamDB compatibility.

11. Self-Certification

Until a formal conformance authority exists, implementations self-certify:

1. Run the conformance suite against your implementation.
2. Publish the resulting report (machine-readable JSON + human-readable summary).
3. State your claimed Tier in the README.
4. Cite the spec version (e.g. "DreamDB v0.1.0").

Conformance and performance are reported separately. The Tier in step 3 is a pass/fail conformance claim (it covers only the conformance criteria of §4.4). An implementation MAY additionally publish the performance profiles (0023-performance-profiles.md) it meets, each with the workload and environment it was measured under. A profile that is unclaimed or unmet never affects the conformance Tier — an implementation can be fully conformant while meeting zero performance profiles.

The community SHOULD maintain a public list of self-certified implementations and their reports. Discrepancies between claimed and actual conformance can be challenged via the public report; this is the same model used by Web Platform Tests, IETF interop reports, and similar.

A formal certification authority MAY emerge in v0.1+; v0 leans on community-maintained transparency.

12. Compatibility Across Spec Revisions

DreamDB spec versions follow semantic versioning:

• Patch (v0.1.x → v0.1.y): clarifications, OQ resolutions, conformance test additions. Existing implementations remain conformant.
• Minor (v0.1 → v0.2): feature additions (new modalities, new algorithms, new optional verbs). Existing implementations remain conformant for the subset they support; new features are opt-in.
• Major (v0 → v1): breaking changes. Implementations MAY support multiple major versions in parallel via a version-prefixed namespace (TBD in v1).

Within v0:

• v0 freezes the data model, address grammar, time encoding, hash function, and HTTP contract.
• v0.1+ MAY add: new spatial-indexing algorithms (dreamdb.pq-ivf, dreamdb.lsh-l2, dreamdb.learned-mlp-v1), additional modalities (AV1, HEVC, FLAC, sensor types), federation primitives, named tags, push-style notifications.

13. Open Questions Resolved by This Document

This document explicitly resolves:

• OQ-11 (concrete CBOR tag numbers): § 3.2.
• OQ-12 (multihash algorithm tags — IPFS-aligned): § 3.1.
• OQ-17 (concrete value for the time-anchor CBOR tag): § 3.2 — folded into local dreamdb.tag = 65521, by application agreement on number and inner type; not used inside DreamDB schemas or claimed globally registered.

The category taxonomy answers the original requests to specify what conformance must cover. It does not prove corpus completeness, an executed result or independent-implementation agreement. Capability-specific unfinished vector questions remain in their owning specs and in INDEX.md.

14. Open Questions Not Resolved

Two operational decisions deferred to community / operator coordination:

• OQ-42 (→ community / future spec): Public registry of conformance reports. v0 is self-certification only; a formal authority may emerge.
• OQ-43 (→ community / future spec): Publication cadence for the conformance suite repository. v0 is unscheduled — releases tied to spec revisions.

End of v0 specification. Implementations interested in v0 conformance should start with 0000-overview.md, build forward through 0001–0008, and verify against the test categories in this document.