DreamDB Specification — 0005: Backend Interface (HTTP Contract)

Status: Draft. Builds on 0000-overview.md, 0001-data-model.md, 0002-content-addressing.md, 0003-time-encoding.md, and 0004-spatial-indexing.md. This document fixes the HTTP semantic contract between the Storage Connector (in-SDK) and the Cloud Object Store (the backend). It resolves OQ-3 (consistency), OQ-22 (local-FS Connector), and OQ-29 (HTTP version requirement). It does not define a language-level interface — the contract is HTTP itself.

1. Purpose

Per 0000 §3, DreamDB's architecture has a clean HTTP boundary between the Storage Connector and the Cloud Object Store. This document specifies what that boundary requires:

• The HTTP verbs the Connector uses and the backend MUST handle correctly.
• The conditional headers that distinguish idempotent puts (content layer) from CAS updates (refs layer).
• The status codes the backend MUST return and the Connector MUST interpret.
• The consistency model the protocol assumes — strong read-after-write for content-addressed objects, linearizable CAS for refs, eventual consistency tolerated for prefix listings.
• The HTTP version required to make the billion-scale hot path achievable.

What this document does not define:

• A language-level trait or interface. The contract is HTTP semantics, not Rust/Go/Python types. (Per project_architecture.md.)
• Backend-specific authentication. Connectors translate to whatever the backend uses (AWS SigV4, GCS HMAC, OAuth, presigned URLs).
• Specific TLS configurations. The backend's HTTPS endpoint is whatever the operator runs; the Connector trusts the system's certificate authorities.

DreamDB defines neither the Application above the SDK nor the Cloud Object Store below the HTTP boundary. This document defines the HTTP boundary itself.

1.1 Auth is a v0 deployment gap (advisory)

v0 leans entirely on the backend's native authentication (AWS SigV4, GCS HMAC, OAuth, presigned URLs). This works for single-tenant deployments where one organization owns the backend. It does not work for multi-tenant production scenarios where you want:

• Per-Space write isolation — writers for Space X cannot write to Space Y's paths even if they share a bucket.
• Per-modality access control — sensitive modalities (PII, regulated data) gated separately from public modalities.
• Per-writer audit logging — backend access logs name the auth identity, but DreamDB doesn't tie that to writer-identity in Manifests.
• Capability tokens — short-lived, scoped tokens for browser-based clients or ephemeral workers.

Cloud object stores authenticate at the bucket level, not at path-prefix granularity (S3 IAM policies can simulate prefix-scoped writes via condition keys, but the rules are fragile and hard to audit). A writer with bucket-level write access can write to any DreamDB path on that bucket.

v0 production deployments MUST address this out-of-band, typically by:

• One bucket per Space (the simplest isolation; works at small Space counts).
• A backend gateway that mediates writes (per-tenant auth at the gateway, then forwards to backend with privileged credentials).
• IAM policy with prefix-condition rules (works on AWS S3 with care; brittle).

v0.1+ MAY define a "DreamDB Auth Layer" spec that addresses these concerns at the protocol level — likely involving capability tokens embedded in Manifests, signed writer identities, and per-Space access policies. Until then, treat auth as the operator's responsibility and document the failure mode prominently in deployment guides.

The gateway named above is not a minor variant of "plain S3 + IAM" — it is a distinct architectural component. 0018 (multi-tenant operation) specifies the contract such a gateway implements: capability-token verification, per-Space quota enforcement (429/507), and fair-share scheduling. Those primitives presuppose a trusted, DreamDB-aware mediation layer in the request path; they are not implementable on plain S3/MinIO, which cannot parse a DreamDB Manifest, verify an Ed25519 capability token, or schedule by tenant_id. The 0000 §3 "no DreamDB code below the HTTP boundary" rule is a data-plane property; 0018's enforcement layer is the documented exception (0000 §3, "the optional mediation gateway"). No reference gateway ships in v0 — 0018 is a contract for a component to be built, and what bare S3 does deliver today is isolation-by-construction only (one-bucket-per-Space + native auth + distinct Timeline IDs).

2. Two Conformance Tiers

A backend may implement one or both of:

Every DreamDB backend MUST be at least ContentStore-Conformant. Every ref-supporting backend MUST also be RefStore-Conformant. A backend that supports only the ContentStore tier hosts hash-addressed Spaces only — clients address Manifests by their hash, with manifest distribution handled out-of-band (e.g. shared via another channel).

This split is a direct consequence of 0000 §5.2 (Content layer immutable, Refs layer optionally CAS-mutable). A backend without conditional-write primitives can host the data plane but not the refs plane.

3. ContentStore Tier (Required)

The ContentStore tier provides idempotent put-by-hash, range reads, and prefix listings over a flat path namespace.

3.1 URL path namespace

The Connector issues HTTP requests against URLs of the form:

<base-url> / <dreamdb-path>

where <base-url> is backend-specific (e.g. https://my-bucket.s3.amazonaws.com/) and <dreamdb-path> is the DreamDB address-grammar path defined in 0002:

<timeline-id> / <modality-tag> / <spatiotemporal-key> / <content-hash>
<timeline-id> / <modality-tag> / track / <content-hash>
<timeline-id> / <modality-tag> / index / <content-hash>
manifests / <content-hash>
manifests / index / <content-hash>
genesis / <content-hash>
spatial-index / <content-hash>
refs / <ref-name>

Connectors MUST pass the <dreamdb-path> to the backend exactly as constructed by the DreamDB Protocol. No path rewriting, prefix injection, or escaping is permitted (paths are already URL-safe per 0002 §8 — base32 hashes, base2 spatial keys, hex time anchors).

3.2 PUT — idempotent put-by-hash

To write a content-addressed Object:

PUT /<dreamdb-path> HTTP/2
Host: <backend>
Content-Type: application/octet-stream
Content-Length: <N>

<N bytes of CBOR or raw payload>

Required backend semantics:

• If the path does not exist, the bytes are stored. Backend returns 200 OK or 201 Created.
• If the path already exists with the same bytes (which the content-addressing guarantees, since the path is the hash), the backend MAY: (a) return 200 OK treating it as a no-op; or (b) return 412 Precondition Failed if the Connector sent If-None-Match: *. The Connector treats both as success.
• If the path already exists with different bytes, the situation is impossible by construction (BLAKE3 collision, ~2⁻¹²⁸). Backends that nonetheless detect this MUST treat it as a server error (5xx).

The Connector MAY use If-None-Match: * to make the put strictly create-only and avoid the implicit-no-op case:

PUT /<dreamdb-path> HTTP/2
If-None-Match: *

412 Precondition Failed returned by If-None-Match: * indicates the object already exists — equivalent to a successful no-op for DreamDB's purposes (the content-addressed bytes are already there). Connectors MUST treat 412 in this case as success, not as failure.

S3 and S3-compatible Connectors MUST transparently use multipart upload for large ContentStore Objects rather than relying on S3's 5 GiB single-PUT limit. The HTTP Connector switches at 100 MiB, S3's recommended threshold. Every non-final part MUST be at least 5 MiB, no part may exceed 5 GiB, and an upload MUST contain at most 10,000 parts. If-None-Match / If-Match belongs on CompleteMultipartUpload, where S3 applies the final write precondition; it MUST NOT be moved to initiation or individual part uploads. A failed logical PUT aborts its initiated upload on a best-effort basis, and a retryable completion conflict restarts from a new upload ID.

This rule applies to every large content-addressed Object, including a Manifest. Bucket splitting bounds individual Bucket Objects but does not bound Manifest size: a highly fragmented Dataset may legitimately produce a Manifest larger than 5 GiB. RefStore values remain small and use ordinary conditional PUT.

3.3 GET — full-object or range read

To read a content-addressed Object in full:

GET /<dreamdb-path> HTTP/2
Host: <backend>

Returns 200 OK with the bytes in the body, or 404 Not Found if the object does not exist (e.g. referenced by a manifest but not yet propagated, or never written).

For a partial read (the hot path for Fragments and Buckets per 0004 §7.3):

GET /<dreamdb-path> HTTP/2
Host: <backend>
Range: bytes=<start>-<end>

Returns 206 Partial Content with the requested byte range, or 416 Range Not Satisfiable if the range is invalid.

The <start>-<end> form follows HTTP Range semantics — <end> is inclusive (HTTP convention), differing from DreamDB's internal [start, end) half-open intervals (per 0002 §6.5). Connectors MUST translate: DreamDB bytes:<a>-<b> (half-open) → HTTP Range: bytes=<a>-<b−1> (inclusive).

For complete Objects that can predate a bounded or paged index format, the Connector interface also exposes a streaming full-object GET. A standard native Connector claiming the bounded-memory historical-reader profile MUST deliver that response as bounded chunks without first aggregating the complete Object in a new in-memory buffer. This is a transport property, not a range-read emulation: an origin is permitted to ignore a Range request and return 200, so issuing successive Range requests cannot establish the bound.

The Protocol component above the Connector remains responsible for Object semantics. When it consumes a content-addressed Object incrementally, it MUST:

1. hash every byte of the complete response;
2. retain only the bounded working set required by the operation;
3. compare the final digest with the address; and
4. expose no decoded result until that comparison succeeds.

A Connector implementation that only implements the compatibility default by adapting a full-object GET remains functionally correct, but does not qualify for the bounded-memory historical-reader profile. Standard DreamDB native HTTP, filesystem, memory, retry, and caching Connectors implement or preserve the streaming form directly. A body failure after chunks have been delivered MUST surface to the consumer; a decorator MUST NOT transparently restart and duplicate an already-delivered prefix.

The WASM direct-fetch Connector streams Response.body. The injected-backend ABI exposes optional getStream(path) -> Promise\<ReadableStream | \{ stream, etag?, totalLength? }>; a backend qualifies for the bounded-memory profile only when it implements that method and yields bounded chunks. Existing backends without getStream remain functionally compatible through their complete-byte get, but that fallback is explicitly the unbounded compatibility profile.

Backend MUST support arbitrary byte ranges — DreamDB makes no assumption about chunk-aligned reads. This is true on every modern object store (S3, GCS, Azure Blob, OSS, MinIO).

For independently verified reads (0002 §6.5.4), the Connector fetches the derived bao-outboard/<subject-multihash> Object and the minimal outward- aligned 1 KiB subject span. It MUST pass both byte sequences to Bao verification against the subject's BLAKE3 digest and MUST return only the caller's exact half-open range after verification succeeds. A missing outboard selects the whole-Object hash-and-slice compatibility path; an existing invalid outboard does not. Bao operates on opaque bytes and their content digest — it does not parse a DreamDB Object and does not weaken §8.4's business-logic boundary. Fetching the complete outboard is conformant in v1; fetching only the parent nodes needed for a range is an optional transport optimization (0002 §6.5.4).

New writes of range-readable Object families MUST establish the deterministic outboard after establishing the subject and before the publication durability barrier. Both writes use ordinary ContentStore semantics. An error establishing the outboard makes the logical Object write fail even when the subject PUT has already landed; the orphaned subject is unreachable and is reclaimed by GC.

3.4 HEAD — existence check & metadata

HEAD /<dreamdb-path> HTTP/2
Host: <backend>

Returns 200 OK with Content-Length (and optionally ETag, Last-Modified) if the object exists, or 404 Not Found. Connectors use HEAD for: existence checks before triggering a re-write, size discovery before deciding on range vs. full-GET, and cheap cache validation.

3.5 LIST — prefix-scoped listing

The list operation is the cold-start discovery primitive (per 0004 §7.3 and 0002 §6.3.1) and the admin-scan primitive (operator-level audits, GC walks). It is never on the steady-state query hot path — see §5.3.1 for the Manifest Supremacy doctrine that governs this.

Backends MUST support listing objects whose paths start with a given prefix. The exact protocol verb is backend-specific:

• S3 / S3-compatible (MinIO, R2, B2): GET /?list-type=2&prefix=<prefix>&continuation-token=<token>
• GCS: GET /storage/v1/b/<bucket>/o?prefix=<prefix>&pageToken=<token>
• Azure Blob: GET /<container>?restype=container&comp=list&prefix=<prefix>&marker=<token>
• Aliyun OSS: GET /?prefix=<prefix>&marker=<token> (S3-style)

Connectors MUST translate the protocol's logical "list by prefix" into the backend's specific list call, including pagination via continuation tokens. The backend MUST return:

• A list of object keys (or full paths, depending on backend convention) matching the prefix.
• For each entry: at minimum the key/path. SHOULD also return Content-Length and ETag to enable cache-friendly behavior.
• A continuation token if the result is paginated.

3.5.1 Sort order

Backends MUST sort results in lexicographic byte order over UTF-8 (equivalent to memcmp over the raw byte sequences). All DreamDB addresses are pure ASCII (base32 hashes, base2 spatial keys, hex time anchors, lowercase modality-tag literals — see 0002 §5, §6, §8), so this is unambiguous. Connectors MUST NOT apply any locale-specific collation, Unicode normalization, or case folding to listed keys. Any backend that delivers non-byte-ordered listings is non-conformant; in practice every backend in §10's matrix conforms.

3.5.2 Pagination correctness

The 1000-object default page boundary on S3 (and most S3-compatible backends) is the most common silent-failure surface for cold-start LIST. Connectors MUST follow a strict discipline:

• Iterate to exhaustion. A Connector that returns "the first page" of a list-prefix call to the Protocol layer is buggy. The Connector MUST iterate continuation tokens until the backend signals end-of-list (IsTruncated=false for S3, absent nextPageToken for GCS, empty NextMarker for Azure, etc.) before returning to the Protocol component.

• Pagination is per-request, not per-session. Continuation tokens are bound to the request that generated them. Connectors MUST NOT cache continuation tokens across separate list-prefix invocations.

• Backend-specific exhaustion signals. Each backend uses a different "no more pages" signal. Connectors MUST translate them all to a uniform "exhausted" return to the Protocol layer:

  Unwrap

  Backend	Truncation signal
  S3 / MinIO	<IsTruncated>true</IsTruncated> + <NextContinuationToken>
  GCS	nextPageToken field present in JSON response
  Azure Blob	<NextMarker>...</NextMarker> non-empty
  Aliyun OSS	<IsTruncated>true</IsTruncated> + <NextMarker>

• Cross-page boundary cases. The 1000th object boundary is a known correctness hazard. Connectors MUST handle the case where a single logical list-prefix result spans K * 1000 + r entries for any K, r ≥ 0, including r = 0 (exact multiple of 1000) and r = 1 (one entry on a fresh page). Conformance test vectors covering these cases are required (OQ-30).

• Verify against Manifest where possible. When the SDK has a Manifest available, the DreamDB Protocol layer SHOULD verify list-prefix results against the Manifest's known bucket-list as a cheap sanity check — silently-truncated listings surface as missing entries, easy to detect and re-list.

Page size: backends typically return up to 1000 entries per page (S3 default; GCS configurable up to 1000; Azure up to 5000; OSS up to 1000). Connectors MAY pass the backend's max page size to reduce round trips but MUST NOT rely on a specific page size for correctness — backends sometimes return fewer entries per page than requested under load.

3.6 DELETE — optional, garbage collection only

DELETE /<dreamdb-path> HTTP/2
Host: <backend>

DELETE is optional at the DreamDB protocol level. The protocol has no delete verb (per 0000 §5.2 — corrections are layered, never destructive). DELETE exists only for garbage collection: an operator may choose to drop old, orphaned content-addressed objects that no live manifest references.

Returns 200 OK or 204 No Content on success, 404 Not Found if absent (treat as success), 403 Forbidden if the operator's policy prohibits.

A backend that disables DELETE entirely is still ContentStore-Conformant — DreamDB's content layer is append-only by design. GC is an operator concern, not a protocol concern.

4. RefStore Tier (Optional)

The RefStore tier adds conditional-write semantics for the small mutable pointer namespaces refs/<ref-name> and federation-refs/<ref-name>. Dataset refs remain the only mutable Dataset objects described by 0000 §5.2; Federation Refs are the corresponding coordinator pointers defined by 0012 §3.3. Both namespaces use the same conditional-write and retry-safety rules below.

4.1 Initial create — If-None-Match: *

The create-only snapshot tags in 0008 §4.4 use this named-root tier under refs/tags/, not ContentStore semantics. Their CLOSED CBOR body differs from an ordinary Ref's raw hash. A 412 requires fetching/decoding the existing tag and comparing its target; it is not content-addressed idempotent success by itself. Tags forbid §4.2 updates. Post-body indeterminacy and durability barriers apply as for any named root; the Connector need not decode tags.

To create a ref that does not yet exist:

PUT /refs/<ref-name> HTTP/2
Host: <backend>
If-None-Match: *
Content-Type: application/octet-stream
Content-Length: 33

<33-byte multihash of the manifest>

Returns:

• 200 OK or 201 Created — ref now points to the manifest.
• 412 Precondition Failed — ref already exists; Connector treats this as a CAS conflict and retries with §4.2 semantics.

4.2 Update — If-Match: <etag>

To advance an existing ref to a new manifest:

GET /refs/<ref-name> HTTP/2
Host: <backend>

Returns the current value plus the backend's strong validator (commonly ETag, sometimes x-goog-generation for GCS). Then:

PUT /refs/<ref-name> HTTP/2
Host: <backend>
If-Match: <etag-from-prior-GET>
Content-Type: application/octet-stream
Content-Length: 33

<33-byte multihash of the new manifest>

Returns:

• 200 OK or 204 No Content — ref advanced atomically.
• 412 Precondition Failed — another writer advanced the ref between the GET and PUT. Connector retries with the new state.

This is optimistic concurrency control, not cooperative locking. No writer ever blocks; conflicts are detected and retried.

4.3 Resolve — GET /refs/<ref-name>

GET /refs/<ref-name> HTTP/2
Host: <backend>

Returns 200 OK with the 33-byte multihash body, or 404 Not Found. SHOULD include ETag so the Connector can use it for a subsequent CAS update without an extra HEAD.

4.4 List refs — optional

GET /refs/?prefix=<ref-name-prefix>

Same semantics as §3.5. Useful for "list all branches" or "list all release tags." Optional even within RefStore-Conformant backends.

4.5 Backend-specific conditional-write headers

The HTTP semantics described above use the IETF-standard If-Match / If-None-Match headers. Some backends use proprietary equivalents:

Connectors MUST translate the standard headers above into whichever flavor the configured backend understands. The DreamDB Protocol component above the Connector emits the standard form.

5. Consistency Requirements (resolves OQ-3)

5.1 Content-addressed objects: strong read-after-write

After a successful PUT of a content-addressed object (Genesis, Manifest, Track, Index Page, Bucket Object, Fragment, etc.), any subsequent GET of the same path MUST return the bytes that were PUT. The backend MUST guarantee this strong read-after-write consistency for content-addressed Objects.

This is the modern S3 default (Amazon S3 went strongly consistent globally in December 2020), the GCS default, the Azure Blob default, the Aliyun OSS default, and the MinIO default. A backend that fails this guarantee is non-conformant.

5.2 Refs: linearizable CAS

A successful conditional PUT with If-Match: <etag> MUST be linearizable — no two writers can both succeed against the same ref with the same If-Match value. The backend's conditional-write primitive MUST satisfy this; all backends listed in §4.5 do.

5.3 Prefix listings: eventual consistency tolerated

LIST /<prefix> MAY lag recent writes. A newly-PUT object MAY not appear in a subsequent list-prefix call for some short window (typically milliseconds, occasionally up to seconds on geo-replicated backends; up to ~1 minute observed on multi-region S3 buckets under transient replication lag).

The classic failure mode: a Pipeline PUTs a Bucket Object → 200 OK → notifies the SDK → SDK does list-prefix → empty result set. The Bucket exists and is fetchable by direct GET against its full path, but the prefix listing has not yet propagated.

DreamDB tolerates this only because the protocol design renders list-prefix non-essential for steady-state operation. See §5.3.1.

5.3.1 Manifest Supremacy (doctrine)

When a Manifest is available, the SDK MUST resolve Object addresses through the Manifest — which is content-addressed and strongly consistent (§5.1) — never through list-prefix, which is eventually consistent.

This is the Manifest Supremacy doctrine. It is not a recommendation; it is a correctness requirement for any conformant SDK.

The doctrine has three concrete consequences:

1. The query hot path uses the cached object_index (from the Manifest), not list-prefix. Per 0004 §7.3 and 0002 §7.3.2, every Track Object's object_index enumerates the addresses of its constituent Bucket / Fragment / Index-Page Objects. Once the SDK has the Manifest, every subsequent fetch is a direct GET against a known, content-addressed path — strongly consistent.
2. list-prefix is reserved for two cases, and only those:
   • Cold-start bootstrap — an SDK with no Manifest at all needs list-prefix to discover what's on the backend.
   • Administrative scans — operator audits (orphan-object detection, GC walks, replication checks) genuinely need to enumerate the backend's contents. These are out-of-band of the query path.
3. Any code path that uses list-prefix during steady-state query operation is a bug. Reviewers must reject SDK implementations that re-list to discover newly-written Bucket Objects. New writes are discovered by reading the new Manifest (which references them by hash) and then direct-GETting via the resolved addresses.

Why this works: the writer's PUT-then-publish-Manifest sequence guarantees that if a reader has the new Manifest, then every Object the Manifest references is available to direct GET (§5.1). list-prefix lag is irrelevant — the reader never depends on it for visibility of recent writes.

Standard write sequence that producers must follow to make this work:

1. PUT all leaf Objects (Buckets, Fragments, Index Pages, etc.).
2. PUT all parent Objects (Track Objects, Manifest Index Pages).
3. PUT the Manifest last.
4. (Ref-conformant backends) advance the ref to the new Manifest hash via If-Match (§4.2).

A reader that resolves the new Manifest is guaranteed to find its dependencies live (§5.1). A reader on the previous Manifest is unaffected. Manifest Supremacy makes the eventual-consistency window of list-prefix invisible to steady-state queries.

A Connector MUST NOT depend on list-prefix returning the most-recent-write. The DreamDB Protocol layer MUST NOT issue list-prefix calls during steady-state operation. Together, these constraints render the eventual-consistency window irrelevant — by construction, not by lucky timing.

5.4 What the protocol does NOT require

• Cross-backend consistency. DreamDB makes no claims about replication between backends. The same Manifest hash on backends A and B is the same Manifest if both backends contain the closure of objects reachable from it.
• Read-your-writes for list operations. See §5.3.
• Atomicity across multiple PUTs. Each PUT is independently atomic; multi-object atomicity (e.g. "publish a Manifest and three new Track Objects together") is the writer's concern. The standard pattern is: PUT all leaf Objects → PUT index pages → PUT Track Objects → PUT Manifest last. A reader resolving the new Manifest finds its dependencies live; a reader on the old Manifest is unaffected.

6. HTTP Version (resolves OQ-29)

The Storage Connector MUST implement the HTTP semantics of this document — the verbs, Range requests, conditional writes, and ETag handling of §3–§5 and §8 — over HTTP/1.1 or any later version. Those semantics are transport-version-agnostic: the byte contract and operation semantics are identical across HTTP/1.1, HTTP/2, and HTTP/3, so an implementation is equally conformant on any of them.

The Connector SHOULD negotiate HTTP/2 (or HTTP/3) when the backend advertises it. This is a performance choice, not a conformance one: HTTP/2's stream multiplexing is what lets the read path's parallel ranged GETs (§6.1) reach the low-latency profile. An HTTP/1.1 deployment is fully conformant — it simply cannot claim the latency-sensitive performance profiles of 0023-performance-profiles.md (per 0009 §4.4, performance profiles never gate conformance). HTTP/1.1 is therefore a first-class transport here, not a degraded fallback.

6.1 Why HTTP/2 matters for performance (informative)

(Informative — performance rationale, not a conformance requirement; see 0009 §4.4 and 0023-performance-profiles.md.) The hot path of a billion-scale query (per 0004 §7.3) issues ~16 parallel ranged GETs to fetch matching Bucket Objects. Under HTTP/1.1:

• Each parallel GET requires its own TCP connection and TLS handshake (~30–80 ms each on cold connections).
• Practical client connection-pool limits (typically 6–20 per host) serialize requests.
• The hot path balloons from ~100 ms to ~500–1000 ms.

Under HTTP/2:

• All 16 GETs share one TCP+TLS connection.
• Streams are multiplexed at the protocol level — no head-of-line blocking at the TCP layer (HTTP/3 fixes this further by moving multiplexing to UDP/QUIC).
• Hot path stays at ~50–100 ms.

6.2 Backend support (informative)

(Informative.) HTTP/2 is widely available at the origin, but end-to-end HTTP/2 is not guaranteed — and, per §6, not required. The S3 REST API negotiated HTTP/1.1 for much of its history (HTTP/2 was the website / CDN-fronted path, not the core object API); and SigV4 signing proxies, on-prem enterprise gateways, corporate MITM proxies, and load-balancer-fronted MinIO frequently downgrade to HTTP/1.1 regardless of what the origin supports. A Connector MUST therefore work over HTTP/1.1 and treat HTTP/2 as an opportunistic upgrade. Approximate origin-side availability:

HTTP/3 (QUIC) is increasingly available; Connectors MAY prefer it when both peers support it. CDN-fronted deployments routinely terminate HTTP/3 at the edge.

6.3 Connector behavior

The Connector:

• SHOULD attempt HTTP/2 via ALPN during the TLS handshake, and SHOULD prefer HTTP/3 when both peers support it.
• When it negotiates HTTP/2 (or HTTP/3) it MUST share a single connection per backend host across concurrent DreamDB queries (a connection-per-request implementation defeats multiplexing — see §8.1), and MUST use the protocol's flow control to avoid application-layer head-of-line blocking. (These MUSTs govern how to use h2/h3 correctly once chosen; they do not require choosing it.)
• When it falls back to HTTP/1.1 it SHOULD use a bounded connection pool per host to parallelize the ranged GETs. Correctness is unaffected; latency is higher (§6.1), so this deployment is outside the latency profiles of 0023.

7. Status Code Reference

A summary of status codes the Connector MUST handle:

8. Connector Responsibilities

The Storage Connector is intentionally minimal — its only job is HTTP transport. But the v0 spec assigns it a small set of well-defined responsibilities:

8.1 Connection pooling and multiplexing

When the Connector negotiates HTTP/2 (or HTTP/3), maintain a single connection per (backend host, auth identity) pair; all concurrent DreamDB requests against that host go through its multiplexed streams (a connection-per-request implementation defeats the multiplexing §6.1 relies on). Over HTTP/1.1, use a bounded connection pool per host instead. Either way, do not open one connection per request. This is a performance discipline for the latency profiles (0023), not a conformance requirement — see §6.

8.2 Retry & timeout policy

• 5xx and 429: exponential backoff, max ~3 retries by default. Configurable.
• 412 on ContentStore PUT: treat as success (the bytes are there).
• 412 on RefStore PUT: treat as CAS conflict, return to caller for retry with new state.
• 404: surface to caller; do not retry.
• Network errors — connect/send failures and dropped connections: timeout, connection reset/refused, no route to host, network/host unreachable, DNS resolution failure, broken pipe, and stale keep-alive ("connection closed before message completed"): retry with exponential backoff. This is safe because GETs are idempotent and ContentStore PUTs are content-addressed (the path is the hash, §3.2), so a connect- or send-phase failure can always be re-issued. A Connector MUST classify these as retryable even when the transport library does not flag the underlying cause (e.g. a send-phase "no route to host" is not always reported as a connect error).
• Default timeouts for GETs: 30 s connect, 30 s from request-sent to response-headers, and 30 s of idleness between body chunks. There is no full-body total deadline by default: such a deadline covers the response body, so on a GET it bounds object-size/throughput rather than health, and a Track Object large enough becomes unreadable however healthy the transfer is. Bounding lack-of-progress instead is size-independent. A Connector MAY offer an explicit total deadline as opt-in.
• Default timeouts for PUTs: 30 s connect, 30 s of no progress while the request body is being handed to the HTTP stack, and 5 min from the last body frame to response headers arriving. There is no total deadline by default, for the same reason as GET: a total deadline bounds object-size/throughput rather than health, so a Manifest, Bucket or Fragment large enough becomes unwritable however healthy the transfer is. The 5 min figure is retained from the previous wording but now bounds only the post-body wait, which does not scale with object size. A Connector MAY offer an explicit total deadline as an additional cap layered over these bounds; where one is configured, whichever bound expires first applies, and its expiry MUST be reported as a configured deadline rather than as a network fault, and MUST NOT be retried automatically.
• Write-path indeterminacy: a PUT that fails once its request body has been fully handed over may already have been applied by the backend, whatever the failure looked like at the transport — a timeout, a dropped connection and a malformed response are indistinguishable in that respect. Until the declared Content-Length has been delivered the backend cannot have completed the write, so earlier failures remain ordinary retryable transport faults. Afterwards the store decides:
  • ContentStore: re-issuing is safe, because the path is the content hash (§3.2), so a retry that races the Connector's own earlier write finds identical bytes and the resulting 412 is success. These remain retryable.
  • RefStore (creates and CAS updates alike): the ref name is not its content, so the same 412 carries no such guarantee. A Connector MUST NOT auto-retry, MUST NOT report the failure as a CAS conflict or as already-exists, and MUST surface an indeterminate result that the caller resolves by re-reading the ref.

8.3 Authentication translation

The Connector receives DreamDB-relative paths from the Protocol component above; it adds the backend's auth (signing, presigned URLs, OAuth tokens, etc.) before sending the request. The DreamDB Protocol component is auth-unaware.

8.4 No business logic

The Connector MUST NOT:

• Parse DreamDB Object bytes (Manifests, Track Objects, Index Pages). These are opaque to the Connector.
• Make decisions about which paths to fetch or in what order. That's the Protocol component's job.
• Cache DreamDB Objects beyond what's needed to fulfill in-flight HTTP responses. Application-level caching belongs in the Protocol component.

The Connector is purely transport. Anything beyond transport is a bug at the Connector layer.

8.5 ETag handling and normalization

ETags are the most variably-formatted field in the HTTP contract — different backends quote them differently, occasionally use weak validators, and use semantically-different inner values (some are hashes, some are opaque generation tokens, some are length-of-bytes-of-hash-of-parts). Connectors are responsible for normalizing ETags so the DreamDB Protocol layer above sees a uniform representation, while preserving the exact bytes needed for round-trip CAS (If-Match) requests.

Two rules govern Connector ETag handling:

8.5.1 Opaque round-trip for If-Match

For ref-update CAS (§4.2), the Connector receives an ETag header on a GET /refs/<name> response and uses it on a subsequent PUT /refs/<name> with If-Match. The Connector MUST round-trip the ETag bytes exactly as received — same quoting, same weak/strong prefix, same content. The Connector MUST NOT interpret, hash-compare, or modify the value. This guarantees the backend's CAS check works correctly regardless of backend-specific ETag semantics.

GET /refs/main                       → ETag: "abc123def456"
PUT /refs/main  If-Match: "abc123def456"   ← exact bytes from GET response

8.5.2 Canonical form for Protocol-layer comparison

When the DreamDB Protocol layer needs to compare or cache ETags across requests (e.g., to detect that a ref has changed since the last GET), the Connector MUST present ETags in a canonical form with these normalizations applied:

• Strip surrounding double-quotes. "abc123" → abc123.
• Strip leading W/ (weak validator prefix). W/"abc123" → abc123.
• No whitespace inside or outside the value. Trim if present (defensive; should not occur on conformant backends).
• No Unicode normalization. ETags are ASCII per HTTP RFC 7232; any non-ASCII byte is a backend bug.

When the Protocol layer hands a canonical ETag back to the Connector for a CAS request, the Connector reapplies the appropriate quoting for the target backend (typically "<value>" for strong ETags). This is symmetric: opaque on the wire, canonical at the Protocol-layer boundary.

8.5.3 What ETags are NOT

• Not content hashes. S3 returns hex-MD5 for single-part objects, hex-of-md5-of-parts plus -N suffix for multipart. GCS returns opaque generation tokens. Azure returns opaque tokens. DreamDB Connectors MUST NOT assume an ETag equals BLAKE3-of-bytes — even though the path is the BLAKE3 hash, the ETag is a separate backend-specific identifier.
• Not stable across regions. Some replication setups change ETags during cross-region copy. Don't compare ETags across backends.
• Not a substitute for * in If-None-Match. If-None-Match: "<some-etag>" checks "object's current ETag is not this value" — different from If-None-Match: * which checks "object does not exist." DreamDB uses * for create-only PUT (§3.2, §4.1); never use a specific ETag in If-None-Match.

Conformance test vectors (0009) MUST include round-trip cases across backends with different ETag conventions: quoted vs. unquoted, weak vs. strong, multipart-style with -N suffix.

9. Local-Filesystem Connector (resolves OQ-22)

HTTP is the remote contract, not a ban on local adapters. The reference dreamdb-connector crate implements FsBackend in src/fs.rs. Its in-process operations must preserve content immutability, byte-range semantics and the linearizable Ref CAS contract; it need not expose an HTTP server. This resolves OQ-22's implementation question and supersedes the original HTTP-only delivery plan. Backend availability in each language binding is a separate API matter.

An HTTP fixture is still required for claims about HTTP headers, status codes or transport behavior; a passing filesystem test cannot establish those claims. Conversely, HTTP success alone does not establish filesystem concurrency or case-sensitivity behavior. Neither atomic rename nor create-only file creation alone is sufficient evidence for replace-if-current CAS.

For local CI, the recommended setup is:

docker run -p 9000:9000 minio/minio server /data

The Connector is configured with http://localhost:9000 as the backend; everything else is identical to production.

10. Backend Compatibility Matrix

△ = partial / region-dependent / preview availability.

The Connector is responsible for normalizing backend differences (proprietary headers, list-API shapes). The DreamDB Protocol layer above sees a uniform contract.

11. Worked HTTP Examples

Concrete request/response pairs for the four most common operations.

11.1 PUT a Manifest

PUT /manifests/blake3-1b3w4q9j5k6f7p8x9z2c4v6n8m1q3w5e7r9t1y3u5i7o9p1a3s5d7f9g1h3j5k HTTP/2
Host: dreamdb-bucket.s3.amazonaws.com
Authorization: AWS4-HMAC-SHA256 Credential=...
If-None-Match: *
Content-Type: application/cbor
Content-Length: 4231

<4231 bytes of deterministic-CBOR Manifest>

HTTP/2 201 Created
ETag: "1b3w4q9j5k6f7p8x9z2c4v6n8m1q3w5e7r9t1y3u5i7o9p1a"
Content-Length: 0

(412 Precondition Failed would mean "already exists" — equivalent to success for ContentStore.)

11.2 GET a Bucket Object byte range (hot path)

GET /xy7g.../embedding.f32.dim=768.bucketed.spatial_bits=18/101100110011010101/blake3-9k8j7h6g5f4d3s2a1q3w5e7r9t1y3u5i7o9p1a3s5d7f9g1h3j5k HTTP/2
Host: dreamdb-bucket.s3.amazonaws.com
Authorization: AWS4-HMAC-SHA256 Credential=...
Range: bytes=3831296-3834367

HTTP/2 206 Partial Content
Content-Length: 3072
Content-Range: bytes 3831296-3834367/12582912
ETag: "9k8j7h6g5f4d3s2a..."

<3072 bytes of f32 vector>

Note: DreamDB bytes:3831296-3834368 (half-open) becomes HTTP Range: bytes=3831296-3834367 (inclusive). Connector handled the off-by-one.

11.3 List bucket addresses for cold-start discovery

GET /?list-type=2&prefix=xy7g.../embedding.f32.dim=768.bucketed.spatial_bits=18/1011 HTTP/2
Host: dreamdb-bucket.s3.amazonaws.com
Authorization: AWS4-HMAC-SHA256 Credential=...

HTTP/2 200 OK
Content-Type: application/xml

<ListBucketResult>
   <Contents>
      <Key>xy7g.../...spatial_bits=18/101100110011010101/blake3-9k8j...</Key>
      <Size>12582912</Size>
   </Contents>
   <Contents>
      <Key>xy7g.../...spatial_bits=18/101101110011010101/blake3-3a4f...</Key>
      <Size>11456001</Size>
   </Contents>
   ...
</ListBucketResult>

The Connector parses the XML (or JSON for non-S3 backends) and returns a list of (path, size) tuples to the Protocol layer.

11.4 Conditional ref update (CAS)

GET /refs/main HTTP/2
Host: dreamdb-bucket.s3.amazonaws.com

HTTP/2 200 OK
ETag: "abc123def456..."
Content-Length: 33

<33-byte multihash of M_old>

PUT /refs/main HTTP/2
Host: dreamdb-bucket.s3.amazonaws.com
If-Match: "abc123def456..."
Content-Type: application/octet-stream
Content-Length: 33

<33-byte multihash of M_new>

Success:

HTTP/2 200 OK
ETag: "ghi789jkl012..."

CAS conflict (another writer advanced the ref):

HTTP/2 412 Precondition Failed

The Connector returns the conflict to the DreamDB Protocol layer; the Protocol retries with the latest state.

12. Out of Scope for this Document

• Authentication mechanisms. Backends use whatever auth they use; Connectors translate.
• TLS configuration. System certificate authorities, OS trust stores. Standard HTTPS practice.
• Quotas, rate limits, billing. Operator concerns; Connector retries on 429 per §8.2.
• Server-side encryption keys (SSE). Backend-specific; configured at the bucket level, transparent to DreamDB.
• Replication, backup, cross-region. Operator concerns; DreamDB is content-addressed, so replication is just "copy the bytes."
• The semantic-cache layer in the DreamDB Protocol component that holds Manifests, Track Objects, and SpatialIndex Objects across queries. That's a 0006 (protocol operations) concern; this document only covers the Connector.

13. Open Questions Surfaced by This Document

• OQ-30 (→ 0009 §6 / §7): Concrete conformance test vectors — HTTP request/response pairs. Resolved: full battery in 0009 §6 (HTTP semantics) and §7 (verb behavior). Test vector battery covers:
  • 412-on-existing-PUT round-trip (treat-as-success vs. treat-as-CAS-conflict per tier).
  • Range-read half-open-vs-inclusive translation (bytes:128-3200 DreamDB-side ↔ Range: bytes=128-3199 HTTP-side).
  • CAS conflict and retry on RefStore PUT against a moving target.
  • List-prefix pagination boundary at exactly K × 1000 and K × 1000 + 1 entries, with continuation-token round-trips for K = 0, 1, and 2 (the 1000-, 2000-, and 3000-object boundary cases). Tests that the Connector returns ALL entries, not just the first page.
  • List-prefix sort order verification: backend MUST return results in lexicographic byte order; test vectors include adversarial keys (paths with adjacent base32 characters, numeric-looking suffixes) to catch locale-collation bugs.
  • ETag flavor compatibility: quoted ("abc123"), weak (W/"abc123"), multipart-style ("abc123-5"). Connector must round-trip opaquely for If-Match (§8.5.1) and canonicalize for cross-request comparison (§8.5.2).
  • Manifest Supremacy assertion: a steady-state query test that asserts NO LIST HTTP request is issued during the hot path. Bootstrap-only LIST is allowed and verified separately.
• OQ-31 (→ v0.1 spec): Multipart upload for large Objects. Partially resolved for S3 / S3-compatible backends by §3.2 after issue #27 demonstrated that Manifest size is not bounded by the Bucket splitting rule. Azure's block-upload boundary remains open.
• OQ-32: RESOLVED as an adapter pattern. The browser SDK's PresignedBackend (dreamdb-dataset-wasm/js/browser-extras.mjs) delegates URL minting and Ref commits to caller-supplied endpoints. Its existence does not standardize an authorization service or relax content/Ref semantics. This is distinct from the unimplemented multi-tenant mediation gateway in 0018.

Next: 0006-protocol-operations.md — defines the verbs (append, layer, query, stream) at the DreamDB Protocol level. Builds on this document's HTTP contract by composing those verbs into HTTP request sequences.