Spec 0014 — Streaming Extensions (Item Chunking + Adaptive Bitrate)

Status: Draft (Phase 3 design). Path A compatibility: §§2.1–2.2 specify the Schema policy and map encoding already emitted by the reference implementation (including python-v0.0.7). The earlier positional-array proposal was not that implementation's wire format; it is withdrawn, not a second accepted encoding. Existing Objects and tags are unchanged. Draft status and the remaining Path A conformance work (OQ-58) do not authorize a reader to reinterpret these published bytes. Depends on: spec/0001, spec/0002, spec/0007. Motivation: spec/0007 §5 defines per-Fragment CMAF encapsulation for media but treats each Item's media payload as a single content-addressed Object. This is correct for small-to-medium clips (the UCF-101 demo proved it for 405 videos averaging ~5 MB). It is structurally inadequate for two production workloads:

1. Large single Items. A multi-gigabyte scan or recording can exceed a backend's single-PUT limit or make whole-Item retry expensive. Backend multipart upload (0005) and content-defined Item chunk storage solve different problems. Chunk storage allows shared chunk bytes to deduplicate; it does not itself remove whole-Item buffering from a writer's API.
2. Adaptive-bitrate playback. A single-rendition Track gives every viewer the same bitrate; production video pipelines emit 3–5 renditions (240p/480p/720p/1080p/4K) so that a client on cellular gets a watchable stream while a client on fiber gets full quality. spec/0007's single-Fragment-per-Item layout has no notion of renditions.

This spec defines two layered solutions:

• Path A — Item chunking: One logical Item becomes N content-addressed chunk Objects plus a small ItemManifest Object that lists them. Per-field opt-in via chunk_size. Enables finer-grained storage and dedup; bounded ingest memory and resumability require corresponding writer behavior and are not implied by this layout. Applies to Image, Audio, AND Video.
• Path B — Adaptive bitrate (CMAF/HLS): An immutable RenditionPlaylist snapshot over aligned, already-published VideoItem Objects. Video-specific.

The two compose: chunking handles per-rendition storage; renditions handle per-bitrate switching. Path B's snapshot contract is defined in §3; the old phase-order plan is not an implementation-status assertion.

1. Purpose

DreamDB's strict immutability and content-addressing make small Items trivial (one Object, one PUT, one GET). Large Items and multi-rendition video need the same immutability guarantees with finer-grained storage units. This spec adds those units without disturbing the address grammar, the modality grammar, or the Track Object index.

By the end of this document the following are concrete:

• Path A — per-field Schema chunk_size policy, ItemManifest ObjectKind, FragmentEntry extension (is_manifest flag), reader stitching logic.
• Path B — a canonical RenditionPlaylist Object, named registry binding, HLS-compatible manifest emission, and enforced cross-rendition segment alignment.
• Address-grammar impact — new items/<hash> and playlist/<hash> slots under the per-Timeline tree.
• Backwards compatibility — datasets with chunk_size = None and no playlist binding produce byte-identical FragmentEntries and Track Objects to today; existing v0 hashes do not change.
• Range-request semantics — Service-Worker / SDK-side range stitching across chunks; per-rendition segment endpoints generated for an HTTP shim.

What stays defined elsewhere:

• Per-Fragment CMAF byte layout — spec/0007 §5.
• Time-bucket placement and Track Object index — spec/0002, spec/0007.
• The address-grammar slots that already exist (init/, vectors/, etc.) — spec/0002.

What this document does NOT define:

• Live streaming (continuous tail-of-history HLS / DASH). spec/0014 is bounded-stream; live extends in a follow-up.
• DRM. Application policy; protocol-level encrypted Objects are separately defined in 0019, whose reference writer/reader remain unimplemented.
• Authentication formats. Whole-Object hashing and authenticated ranges follow 0002 §6.5.4. An arbitrary slice cannot be authenticated by hashing that slice against the whole Object's digest.
• Transcoding pipelines. Producers are responsible for emitting the renditions; DreamDB stores them.

2. Path A — Item Chunking

2.1 Schema write policy, not a modality discriminator

Image/Audio/Video fields carry optional Schema chunk_size: u64. When absent, the ordinary writer uses a direct Fragment; when present, it writes an ItemManifest even if the payload fits in one chunk. This policy does not add a chunk_size= component to the modality tag. Readers select the representation from each FragmentEntry's is_manifest, never from tag spelling or the current write policy. Existing direct and chunked entries can share a modality.

For compatibility with the shipped writer, the effective chunk size is max(chunk_size, 1) bytes: explicit zero selects one-byte chunks, not disabled chunking. Writers split non-empty payloads at this size, with a shorter final chunk permitted. The stored sizes, not Schema, determine a reader's offsets. The former 64 KiB–16 MiB interval is sizing advice, not a wire acceptance range; small chunks and larger valid chunk sizes MUST NOT be rejected merely for lying outside that interval. Backend request and implementation resource limits still apply and must be reported explicitly, not silently truncate the requested size.

2.2 The ItemManifest Object

When chunking is enabled, each Item's blob produces N content-addressed chunk Objects plus one ItemManifest Object that enumerates them.

Address path:

<timeline>/<modality>/items/<itemmanifest-hash>

Canonical CBOR encoding (CLOSED maps; each named key occurs exactly once):

{
  "total_size": <u64>,
  "chunks": [
    {"size": <u64>, "hash": <33-byte multihash>},
    {"size": <u64>, "hash": <33-byte multihash>},
    …
  ]
}

Constraints:

• chunks MUST be non-empty. Positional-array manifests and positional-array chunk entries are not alternative encodings of these maps.
• Sum of chunk sizes MUST equal total_size as a mathematical u64 sum, not a wrapping sum. Sizes describe the actual referenced Object bytes.
• Chunks listed in byte order; readers concatenate in this order to reconstruct the blob.
• Empty Items (zero-byte blobs): the writer emits total_size = 0 and one chunk with size = 0 and the multihash of the empty byte string. That empty chunk Object is stored and reachable like any other chunk. An empty chunks array is invalid, including when total_size = 0. Readers use declared sizes; they do not impose the writer's current chunk-size policy on historical data.

2.3 Chunk Object storage

Each chunk is a content-addressed Object at the existing time-bucketed Fragment path:

<timeline>/<modality>/<time-bucket>/<chunk-hash>

Same path grammar as a direct Fragment; chunk Objects use time-bucket 0 (encoded per 0003), independently of the Item's anchor. This lets a reader reconstruct their paths from the ItemManifest's hashes without an unstored bucket choice. Within a timeline/modality, equal chunk bytes give the same address and create-only PUTs deduplicate them. Existing chunks are not relocated.

2.4 FragmentEntry extension (Track Object index)

Per spec/0002 §7.3, the Track Object's object_index carries one entry per Item. Existing 4-tuple positional CBOR:

[t_start, t_end, byte_size, fragment_address]

Path A extends this to a 5-tuple ONLY when chunking is enabled for the field:

[t_start, t_end, byte_size, fragment_address, is_manifest]

The 5th element is a boolean:

• false ⇒ fragment_address points at a single chunk Object (the legacy case). Track Objects of unchunked modalities continue to emit 4-tuples and produce byte-identical encodings to today — CRITICAL for content-addressability of existing datasets.
• true ⇒ fragment_address points at an ItemManifest Object. The reader fetches it, then range-fetches the listed chunks.

Encoder rule: emit 4-tuple iff is_manifest = false; emit 5-tuple iff is_manifest = true. Per spec/0002 §3.1.1, readers MUST accept both lengths and treat missing trailing fields as defaults. A v0 reader that knows only 4-tuples ignores the 5th element and decodes a chunked Track as a regular single-fragment Track whose Fragment is the ItemManifest CBOR bytes. It therefore returns the index Object in place of the Item's content. This is a wrong result, not a degraded one, and it is a defect of the original specification rather than a compatibility property.

The reachability consequence is worse than the read consequence. The chunk Objects are named only inside the ItemManifest, so a collector that does not follow is_manifest marks the ItemManifest and never reaches the chunks; a subsequent sweep deletes live chunks and leaves an ItemManifest pointing at Objects that no longer exist.

is_manifest is therefore a critical extension under 0002 §3.1.0, changing both payload interpretation (condition 1) and reference closure (condition 3) — the latter without introducing any new reference location, by changing what fragment_address points at. It is a named historical exception under 0002 §3.1.0.1: the encoding remains a valid format that MUST NOT be reinterpreted or require rewriting, an implementation declaring chunked-Item support MUST implement the semantics above, and an implementation without that capability MUST refuse explicitly rather than resolve Items from, or sweep against, a Manifest that uses it. The per-role obligations and the rollback rule are in 0002 §3.1.0.3.

2.5 Read path — stitching

When the SDK fetches an entry with is_manifest = true:

1. Fetch the ItemManifest Object at fragment_address.
2. Decode and validate the CBOR; obtain the chunk list and total_size. The declared chunk sizes MUST sum to total_size before either read mode is served.
3. If the caller requested a valid byte range [A, B) within the Item: a. Identify which chunks i ∈ [start_chunk, end_chunk] overlap [A, B). b. For each overlapping chunk, range-fetch its bytes (full chunk if fully covered; partial range if partial). c. Concatenate in chunk order.
4. If the caller requested the whole blob: range-fetch all chunks, concatenate.
5. Verify the result length for the selected read mode:
   • whole-Item read: the concatenated byte count MUST equal total_size;
   • range read [A, B): the concatenated byte count MUST equal B - A.

A mismatch is a critical error (a listed chunk or returned range is shorter/longer than declared) and surfaces as protocol corruption. The range case MUST NOT compare its result with the whole Item's total_size; doing so would reject every proper subrange. These length checks are additional to the content-address and byte-range integrity rules in 0002 §3.1.2 and §6.5.4; they do not weaken or replace them.

Examples for an Item with total_size = 4096:

• A whole-Item read returns 4096 concatenated bytes and compares 4096 with total_size.
• A valid [0, 1024) read returns 1024 concatenated bytes and compares 1024 with 1024 - 0, not with 4096.

Chunk fetches MAY be issued in parallel (HTTP/2 multiplexing). The SDK SHOULD use a small concurrency cap (4–8) to avoid head-of-line blocking in the connector pool.

2.6 Service-Worker streaming pattern

For browser playback of large chunked video, a Service Worker is the natural integration point:

On fetch('/item/<manifest-hash>'):
  1. Fetch + cache the ItemManifest (small, immutable).
  2. Parse the browser's Range header.
  3. Map the range to chunks; issue per-chunk range fetches against the backend.
  4. Stitch and return 206 Partial Content with the correct Content-Range.

The browser's <video> element issues range requests against this Service-Worker-intercepted URL; the SW transparently materializes the byte range from the chunks. Pseudocode in the existing chunking plan; conformance tests for the stitching logic ship in spec/0009.

2.7 GC reachability

The reachability walk (spec/0006 §7.3.1) for chunked Tracks:

For each Track in reachable manifests:
  For each FragmentEntry in Track's object_index:
    If is_manifest = true:
      Mark fragment_address (ItemManifest) reachable.
      Fetch ItemManifest; decode.
      For each chunk's hash:
        Mark chunk Object reachable.
    Else:
      Mark fragment_address (chunk Object) reachable.

Without the transitive walk, GC could DELETE chunks still referenced by live ItemManifests, producing dangling references and silent data loss. The conformance suite (spec/0009 §7) MUST include a chunked-Track GC test.

3. Path B — Adaptive Bitrate (CMAF Renditions)

3.1 Snapshot model and address modality

Every rendition is first published through an ordinary FieldKind::VideoItem field. Those field bindings and their VideoItemTrack indexes remain authoritative for ingest and current-item lookup. Publishing a playlist does not replace, wrap, or mutate those Tracks. Instead it pins one already-published VideoItem Object from each field into an immutable snapshot.

The playlist Object has its own canonical address modality:

ai.dreamlake.media.rendition_playlist.cmaf.renditions=<N>.version=1

N MUST equal the number of entries in the Object and MUST be in [2, 16]. There is no renditions=N parameter on a VideoItem modality. The earlier design that placed such a parameter on a video Track and made a playlist replace its registry binding was never implemented and has no migration path.

3.2 The RenditionPlaylist Object

A RenditionPlaylist enumerates aligned VideoItem snapshots and the common segment boundaries at which a player may switch between them.

Address path:

<timeline>/<modality>/playlist/<playlist-hash>

CBOR encoding:

{
  "v":                1,
  "item_key":         <bstr>,
  "t_start":          <uint>,
  "duration":         <positive uint>,
  "renditions": [
    {
      "label":           "<string>",                ;; "240p", "1080p", "4K", etc.
      "bandwidth":       <unsigned int>,            ;; bits per second; for client selection
      "resolution":      [<width: uint>, <height: uint>] | null,   ;; null when unspecified
      "codec_string":    "<RFC-6381 codec ID>",     ;; e.g. "avc1.4d401e"
      "modality":        "<VideoItem modality>",
      "item":            <multihash-of-VideoItem-Object>,
    },
    …
  ],
  "segments":          [[<relative-start>, <relative-end>], …],
  "default_rendition": <uint>,                     ;; index used only when no label is supplied
}

This is a CLOSED canonical-CBOR map. Each rendition map is also CLOSED. Labels are unique safe tokens, bandwidth and any resolution dimensions are positive, and default_rendition is in range. segments is non-empty, strictly positive, contiguous, and covers exactly [0, duration).

3.3 Publication and read-time alignment

Before writing the playlist Object or advancing a Ref, a publisher MUST resolve each named VideoItem field through the parent Manifest and verify all of the following:

1. The requested item_key exists in every field.
2. Every selected item has the same absolute t_start and duration.
3. Every selected item has the same ordered list of relative Fragment boundaries.
4. The common boundaries are the canonical segments stored in the playlist.

The publisher does not rewrite any media Object. Anchor-aligned renditions are produced by running the same CMAF segmenter and keyframe schedule over the same source at different encode parameters.

A reader MUST resolve each rendition's declared VideoItem modality from the retained Manifest registry, fetch and hash-verify the pinned VideoItem Object, and independently verify that its actual Fragment boundaries equal segments. A requested range MUST begin and end on common segment boundaries. Therefore switching changes only the selected rendition between segments; it never combines approximate boundary metadata with different media bytes.

Stream selection (0006 §4.4.2) uses these same bindings and alignment checks: an omitted label selects default_rendition, an explicit unknown label fails, and invalid default metadata is refused rather than repaired or guessed. The Dataset media Stream emits verified init followed by on-demand Fragments; it neither transcodes nor reselects a rendition while reading a pinned snapshot.

3.4 Manifest registry reference

A named playlist is bound in the Manifest registry independently of the VideoItem field bindings:

"registry": {
  "dreamdb.rendition_playlist.<name>": {
    "v":        1,
    "timeline": <multihash>,
    "modality": "ai.dreamlake.media.rendition_playlist.cmaf.renditions=<N>.version=1",
    "playlist": <multihash-of-RenditionPlaylist>,
  }
}

The binding map is CLOSED. The name is 1–64 ASCII alphanumeric, dot, underscore, or hyphen characters. The Timeline and canonical modality in the binding MUST match the Object address.

A collector that understands this capability MUST decode the playlist and traverse every referenced VideoItem's closure: the VideoItem Object, init segment, direct Fragments or ItemManifest, and any chunks. This traversal is required even if later field publications replaced the current item. The playlist is a retained snapshot, not a hint back to the current Track.

3.5 HLS interop emission (optional)

A SDK MAY emit an HLS-compatible manifest from a RenditionPlaylist on demand. This is a translation, not a storage format — the canonical form is the CBOR RenditionPlaylist; HLS .m3u8 is for browser/CDN compatibility.

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=480x270,CODECS="avc1.4d401e"
<URI to rendition-0 sub-playlist>
#EXT-X-STREAM-INF:BANDWIDTH=2400000,RESOLUTION=1280x720,CODECS="avc1.4d401e"
<URI to rendition-1 sub-playlist>
…

Each rendition sub-playlist enumerates the committed common segments. Its relative /<label>/init and /<label>/segment/<ordinal> URIs are endpoints for a Service Worker or HTTP shim that calls the DreamDB read API; they are not raw Object paths and do not presume direct Fragment storage. HLS emission is deterministic metadata translation. It does not rewrite media and HLS is not the canonical stored form.

3.6 Path A × Path B composition

Path B is independent of the selected VideoItem's physical Fragment representation. If Path A stores a selected Fragment through an ItemManifest, the normal VideoItem materializer follows that indirection. The playlist still pins the same VideoItem Object and common segment table; it neither duplicates nor changes the ItemManifest encoding. Cross-rendition byte deduplication is not promised.

4. Address-grammar additions

Two new content-addressed slots under the per-Timeline tree:

<timeline>/<modality>/items/<itemmanifest-hash>     ;; Path A — Item chunk index
<timeline>/<modality>/playlist/<playlist-hash>      ;; Path B — Rendition enumeration

These slots are NEW. spec/0002 §6.3's path parser (4-segment match arms for track/, index/, init/, vectors/, bucket/) extends with two more match arms for items/ and playlist/. spec/0009's conformance suite adds path-roundtrip vectors for both.

5. Backwards compatibility

Datasets created before spec/0014 remain valid and byte-identical.

• Path A: direct entries emit 4-tuples; chunked entries emit manifest-bearing 5-tuples, independent of modality spelling (§2.1). Existing unchunked Track bytes are unchanged. A capable reader observing a 4-tuple treats it as is_manifest = false.
• Path B: absence of a dreamdb.rendition_playlist.<name> registry binding leaves every VideoItem field and Track unchanged. A playlist-capable reader adds named snapshot playback; it does not alter ordinary VideoItem reads.

For a reader without the relevant capability:

• A reader that treats a manifest-bearing 5-tuple as a 4-tuple can fetch ItemManifest bytes as media, producing incorrect output or a decoder error. Historical implementations' exact behavior is not guaranteed. Under the present 0002 §3.1.0 rule it MUST refuse before entering those semantics; trailing-field tolerance is not permission to ignore is_manifest=true.
• Playlist binding: an implementation that does not support this capability MUST refuse before entering playlist semantics or reachability traversal. It may still perform opaque, byte-preserving replication. It MUST NOT silently ignore the binding while claiming a complete semantic copy or running GC over affected retained roots (spec/0002 §3.1.0).

6. Storage and latency at scale

6.1 Path A bandwidth profile

For a 4 GiB Item and chunk_size=4 MiB, payload arithmetic gives 1,024 chunks plus a map-encoded ItemManifest (§2.2). At 100 Mbit/s, payload-only transmission takes about 344 seconds before overhead, whether it is one Object or many. Concurrency does not multiply link bandwidth. Already uploaded chunks can be reused, but cross-process resume requires the writer to preserve or reconstruct progress; the layout alone does not implement it.

An aligned 100 MiB slice spans 25 chunks; an unaligned slice can span 26. The reader can range-fetch boundary chunks under the integrity rules in 0002. A single unchunked Object also supports Range GET, so no inherent “8× more requests” disadvantage follows for it. Compare actual fetched bytes, metadata reads and verification mode before making a performance claim.

6.2 Path B storage cost

A 1-hour 1080p video at 5 Mbps with 4 renditions (240p/480p/720p/1080p) at typical bandwidths (0.4 / 1.0 / 2.5 / 5.0 Mbps):

• Total bytes: (0.4 + 1.0 + 2.5 + 5.0) × 3600 / 8 = ~4.0 GB across all renditions.
• vs single-rendition 1080p: 2.25 GB.
• Storage premium: ~1.8× for adaptive playback. Industry-standard tradeoff.

6.3 Combined Path A × Path B at billion-scale

10⁶ hours of 1080p video, 4 renditions, chunk_size=2 MiB:

• Total bytes: 10⁶ × 4 GB = 4 PB.
• Chunks: ~2 trillion. ItemManifests: ~2 billion.
• Most queries access ~1 hour of one rendition: 720 chunks fetched, ~1.4 GB transferred, streaming concurrent → playback starts in seconds, finishes when video does.

The scale targets exceed v0's single-Timeline ceilings (1B Items max per Track) — production Petabyte-scale deployments combine Path A + Path B with spec/0012 federation, sharding Timelines per-asset or per-day-range.

7. Out of scope

• Live streaming. Continuous tail-of-history HLS (sliding-window playlists, EXT-X-ENDLIST absent) is not in v0.X. The protocol's append-only model handles live ingest; what's missing is producer-side tooling to publish playlist deltas at sub-segment cadence. Defer to v0.X+1.
• Per-rendition independent compression. Renditions all use the same chunk_size in v0.X. Per-rendition tuning is plausible but adds parameter complexity; defer.
• Subtitle / caption renditions. A stream_role: "text" entry in the playlist is the natural extension; defer until needed.
• DASH MPD emission. HLS is the lingua franca; DASH support follows the same translation pattern but is operator-layer.
• Cross-rendition byte dedup. Possible if encoders produce shared chunks across qualities (unlikely at the byte level; produced chunks differ even at matching keyframes due to quantization). No protocol support.

8. Open questions

• OQ-57 (→ this spec): Should ItemManifest contain a top-level mime_type field? Currently the modality tag carries it. Adding to the manifest lets a generic reader serve any chunked blob without knowing its modality, but adds redundancy. Defer until measurement.
• OQ-58 (→ spec/0009): Portable chunked-read, authenticated-range and GC evidence against the map-based ItemManifest contract (§2.2). Unsupported manifest-bearing entries must be refused, not called forward-compatible. The wire/configuration reconciliation does not implement the outstanding portable conformance corpus; this question remains open.
• OQ-59: RESOLVED — existing Fragment path. §2.3 and the reference put_chunked_item use <timeline>/<modality>/<time-bucket>/<hash>, not a new chunks/ slot.
• OQ-60: Should the per-chunk numeric bound be narrower than u64? Canonical CBOR already uses the shortest integer encoding for a value, so calling a value u32 instead of u64 saves no bytes for the same size. A narrower bound would be a validation/compatibility decision, not a fixed four-byte-per-chunk saving; no such narrowing is made here.
• OQ-61: Resolved. 0006 §4.4.2 defines explicit VideoItem/playlist selection, committed default versus exact label, switch boundaries, pinned reads and the lazy media Stream adapter (#360). Unknown labels never fall back; automatic bitrate selection is not promised.

Next: spec/0009 amendment with conformance vectors for the new ObjectKinds (ItemManifest, RenditionPlaylist, VectorCompressor, GraphIndex, GraphPage, FederationManifest).