Spec 0019 — Data-Plane Encryption
Status: Normative format (v0.X). No reference Dataset writer or reader ships yet; implementations MUST NOT claim this capability until they implement this document and pass the 0009 §8.6.5 vectors.
Depends on: spec/0001, spec/0002, spec/0005, spec/0012, spec/0017, spec/0018.
Cryptographic references: RFC 5297 (AES-SIV), RFC 5869 (HKDF), RFC 3394 (AES Key Wrap), FIPS 180-4 (SHA-256).
1. Scope and trust boundary
This document defines encryption of DreamDB's content-addressed data plane. It provides confidentiality and integrity at rest while retaining ciphertext content addresses, range reads, opaque federation and an explicit deduplication boundary.
It does not provide searchable encryption. A semantic SDK decrypts before parsing, indexing or searching, and is inside the plaintext trust boundary. Refs, Genesis Objects and Manifest Objects remain cleartext so an implementation can discover the required capability and key policy before it fetches encrypted Objects. TLS remains the transport requirement (0005).
The feature is not yet implemented by the reference Dataset, Python or WASM SDK. This format authorizes a later interoperable implementation; it is not evidence that current releases encrypt data.
2. Capability boundary
Encryption is a critical extension under 0002 §3.1.0: it changes payload interpretation, and encrypting a reference-bearing Object changes how a closure traversal obtains its references. It MUST NOT be introduced through an ignorable map key.
An encrypted Space therefore uses the CLOSED tracks.form = "lineage-v3" container defined in 0002 §7.2.7. Lineage-v3 is Lineage-v2 plus one required encryption policy. An implementation that does not recognise the form MUST reject the Manifest before it resolves an Item, reads parent state, or marks or sweeps against it. Byte-exact opaque copying remains legal without the capability.
Every content-addressed Object reachable below an encrypted Manifest is an encrypted envelope (§4), including Track, Index Page, ItemManifest, decided-set and leaf payload Objects. Refs, Genesis and Manifest Objects are not envelopes. The deterministic bao-outboard/<envelope-hash> proof metadata of 0002 §6.5.4 is also not an envelope: it is not an ordinary content-addressed DreamDB Object, reveals only the ciphertext tree, and must remain usable before decryption so a range can be bound to its expected envelope address. A semantic reader, a writer that consumes parent state, an auditor, a selective or reachability-based replicator, and a collector all require a usable domain key. An opaque copier needs no key and verifies the envelope's ordinary content address.
Changing between unencrypted and encrypted forms, or changing mode, suite or domain_id, requires a full Reencode whose migrated Manifest is a root (parents = []; 0002 §7.2). A linear successor may change only key_slots. A merge requires identical mode, suite and domain; otherwise it fails. The merge writer chooses the successor's canonical non-empty slot set and MUST verify that at least one selected slot recovers the declared domain key before publication.
3. Encryption domains and recipient slots
3.1 Domain key and identity
An encryption domain owns a uniformly random 32-byte Domain Wrapping Key (DWK). Its public identity is:
domain_id is exactly 32 bytes. A candidate DWK is accepted only when recomputation matches in constant time.
Sharing a DWK deliberately creates one deduplication and compromise domain; that domain may span Spaces. Per-Space isolation means generating a distinct DWK for each Space. domain_id contains no secret-key material, but its preimage-resistance assumes a uniformly random 256-bit DWK; passwords and other low-entropy inputs are forbidden.
3.2 EncryptionPolicy (CLOSED)
The Lineage-v3 encryption value is a CLOSED canonical-CBOR map with exactly:
All five keys are required exactly once; unknown or duplicate keys are malformed. key_slots is non-empty and strictly increasing by (recipient UTF-8 bytes, key_ref UTF-8 bytes, wrapped_dwk bytes). Equal elements are duplicates and are rejected. Array order has no priority meaning.
3.3 KeySlot (CLOSED)
The three keys are required exactly once. The strings are opaque, compared byte-for-byte and MUST NOT be Unicode-normalised. key_ref identifies an operator-configured KMS/KEK operation; wrapped_dwk is that provider's authenticated ciphertext. DreamDB does not claim that cloud KMS ciphertexts share a wire algorithm.
A slot succeeds only when the named provider authenticates it, returns exactly 32 bytes, and those bytes recompute domain_id. Failure of one slot MAY lead to another slot; no failure may produce plaintext or fall back to unencrypted interpretation. At least one slot MUST be usable by a writer before it publishes the policy. A reader with no usable slot returns an authorization/key-unavailable error, not malformed plaintext.
Slots are outside encrypted Object envelopes. Adding, removing or rewrapping slots changes the clear Manifest address but does not change any encrypted Object or its address. Removing a slot is not revocation while a principal, older retained Manifest, cache or other KEK can still recover the DWK.
4. Encrypted Object envelope
4.1 Framing and address
header_len MUST be in [1, 512]. The bound lets a reader inspect the untrusted 12-byte prefix without allowing it to trigger an unbounded header fetch before the prefix is authenticated. All length and offset arithmetic is checked; overflow is malformed. Bytes after the one computed final chunk are forbidden. The normal DreamDB address is the BLAKE3 multihash of the complete envelope, including prefix, header, SIVs and ciphertext. No component receives a second address.
The magic is compared as four bytes, not as a host-endian integer (0007 §2.1). A reference-bearing Object is parsed for references only after its complete plaintext has authenticated.
4.2 Canonical header (CLOSED)
The header is deterministic CBOR and contains exactly:
All six keys are required exactly once. Unknown or duplicate keys, non-canonical CBOR, a mode or suite not named above, a domain_id other than 32 bytes, plaintext_len > 2^40, any other chunk size, or a mode-inconsistent wrapped_key length is malformed. Header mode, suite and domain MUST equal the enclosing Lineage-v3 policy before a key is unwrapped.
4.3 Convergent key bundle
For plaintext M:
AES-KW uses RFC 3394's default initial value A6A6A6A6A6A6A6A6. The 64-byte DEK is the two-key AES-256-SIV key required by RFC 5297.
Sealing P || DEK removes the old sketch's key-recovery cycle without publishing a low-entropy confirmation oracle. A reader first authenticates and unwraps the bundle, derives the DEK again from sealed P, and compares all 64 bytes in constant time before decrypting a chunk. A mismatch is malformed encrypted data. A full-object read also hashes the recovered plaintext and compares it to P before reporting full-object success. A range-only read authenticates its chunk but, by construction, cannot independently recompute the whole-object plaintext hash.
Same plaintext, DWK and v1 parameters produce byte-identical envelope bytes and therefore the same address. Different domains do not. Equality and length leakage inside one domain are intentional properties, not hidden side effects.
4.4 Randomized key bundle
The writer samples a fresh uniformly random 64-byte DEK once per logical Object write:
Two writes of the same plaintext use different DEKs and therefore different envelopes and addresses. A transport retry MUST reuse the already-formed envelope; generating another DEK for a retry produces an orphan and is not the same logical write.
5. Chunk encryption and range reads
The plaintext is divided into consecutive 1,048,576-byte chunks. The final chunk may be shorter. Empty plaintext has exactly one empty chunk.
The 1 TiB plaintext limit bounds chunk_count to 2^20. Each sealed chunk is the RFC 5297 deterministic output:
using AES-256-SIV with the Object's DEK and these three associated-data strings, in this order:
- the exact bytes
"DENC" || u32be(1) || u32be(header_len) || canonical_header; u64be(chunk_index), where the first index is zero;u32be(chunk_plaintext_len).
No nonce is supplied: this is RFC 5297 deterministic authenticated encryption, selected because convergence intentionally repeats ciphertext for equal input. It does not misuse an algorithm whose nonce is expected to be unique.
A reader begins with the expected envelope address carried by the authenticated DreamDB structure that led to this Object. It may inspect an unverified 12-byte prefix only to reject an invalid magic/version or a header_len outside [1, 512]; it MUST NOT use unverified bytes for any other semantic decision. Before parsing or trusting the header, it authenticates [0, 12 + header_len) against that expected address with the bao-outboard/<envelope-hash> procedure of 0002 §6.5.4. It then validates §4.2, unwraps and validates the key bundle, and computes a chunk range. For chunk i < chunk_count - 1:
The final sealed length is (plaintext_len - i * chunk_size) + 16; it is 16 for the empty Object. Before SIV decryption, the reader authenticates the complete sealed-chunk byte range against the same expected envelope address with the same Bao outboard. It then MUST authenticate the whole sealed chunk with AES-SIV before exposing any plaintext from it. The two checks prove different facts: Bao binds the fetched header and chunk to the Manifest-selected ciphertext address; AES-SIV binds the chunk to its header, domain key, index and length. AES-SIV alone is insufficient because a backend could otherwise substitute a self-consistent header and chunk from another valid Object in the same domain.
An absent outboard selects only the whole-envelope hash-and-decrypt fallback of 0002 §6.5.4; it never permits an unproved ranged result. A present but invalid outboard is an integrity failure and cannot downgrade. Truncation, trailing bytes, reordered chunks, a changed authenticated header, wrong position/length, address-proof failure, AES-KW integrity failure or SIV failure is an error. The reader MUST NOT return a prefix from any failed check.
Ciphertext MAY be cached persistently by envelope address. Decrypted chunks are memory-only by default, keyed at least by (envelope_address, chunk_index, domain_id), never shared across domains, and purged when the local key grant is lost or erasure handling runs. This protocol cannot recall plaintext copied outside a conforming cache.
6. Writers, readers, GC and federation
- A writer validates Lineage-v3 and a usable slot before producing any encrypted Object. It wraps and encrypts before hashing, PUTs only envelope bytes under that envelope's address, and publishes the standard Bao outboard before any Manifest can make a range-readable envelope reachable.
- A reader binds whole-envelope or Bao-proved ranged bytes to the expected ordinary content address before trusting the header, then performs the authenticated steps above. It never treats a clear Object as encrypted or an envelope as clear based on a magic-byte guess; the Manifest form decides.
- A collector traverses the clear Manifest, decrypts every reference-bearing reachable Object before following its references, and aborts mark/sweep if any required key or authentication is unavailable. It MUST NOT record that as a skippable gap and continue sweeping.
- A byte-exact federating copier may copy a known complete closure without keys and verify each ciphertext address. Any copier that discovers the closure by semantics, or attests semantic completeness, needs the key. A federated reader needs a usable recipient slot; the protocol does not translate grants between KMS providers.
- Storage quota counts envelope bytes, because those are the bytes stored. Plaintext size is diagnostic metadata, not chargeable storage.
No per-modality opt-out exists. A partially encrypted closure would let a field silently cross the promised boundary and would require a second capability negotiation surface. A future mixed policy is a new tracks form and migration, not an optional registry key.
7. Rotation, revocation and erasure
Rewrapping the same DWK under another KEK changes key slots and Manifest bytes only. Encrypted Object addresses stay stable. Rotating the DWK changes domain_id and requires a full Reencode root.
Domain-level cryptographic erasure requires making every recovery path for the DWK unusable: destroy or revoke all KEKs and grants able to open any slot in every retained Manifest/root, erase live process copies, and purge decrypted caches. Ciphertext may remain. This is an operator action and its guarantee is only as strong as the external KMS deletion and backup policy.
V1 does not promise per-Item or per-subject cryptographic erasure. The wrapped key is immutable inside a content-addressed Object, and one deduplicated Object may serve several Items. Selective shredding would need a separately mutable per-subject key index and a rule for shared Objects; neither exists here. Tombstones remain logical deletion (0020), and physical GC remains the byte-reclamation mechanism.
8. Security analysis
- A backend-only adversary sees ciphertext addresses, envelope sizes, domain IDs, equality within a convergent domain, graph/access patterns and clear Manifest metadata. It does not receive the plaintext hash or a key-confirmation value.
- Randomized mode hides equality across logical writes but not length, access patterns or clear metadata.
- A DWK compromise exposes every Object in its domain. A KMS grant is an authorization boundary only while the recipient has not retained the DWK or plaintext.
- AES-SIV authenticates each chunk and its location/header. It does not authenticate sibling chunks that were not fetched; the normal envelope content hash authenticates complete fetched bytes, and a full convergent read additionally validates
P. - The SDK process is inside the plaintext boundary. Memory disclosure, malicious code in that process, plaintext logs, swap, crash dumps and caller-retained buffers are outside what ciphertext-at-rest can prevent.
- The security of
domain_idand convergent derivation assumes a uniformly random DWK. Human passwords are forbidden. - The opaque
wrapped_dwkis no stronger than its provider's authenticated wrapping, authorization, audit and destruction semantics. A provider that returns unauthenticated or exportable low-entropy key material is not conforming to the slot contract.
9. Conformance
0009 §8.6.5 defines mandatory vectors for an implementation claiming this capability. They fix canonical header/envelope bytes and address, convergent recovery, domain separation, randomized uniqueness using supplied deterministic fixture DEKs, tamper refusal, address-bound chunk range location, empty plaintext, malformed key bundles, and Lineage-v3 policy structure.
Vectors provide keys and KMS results as test inputs; they do not claim to certify a cloud KMS. A production implementation must additionally test its provider adapter. KMS latency, cache-hit rate and throughput are performance/operations evidence, not protocol conformance.
10. Resolved questions and non-goals
- OQ-79: resolved by the sealed
P || DEKconvergent bundle (§4.3). - OQ-80: KEK rewrap preserves Objects; DWK rotation requires Reencode (§7).
- OQ-81: opaque federation needs no key; semantic use requires an operator-provisioned slot. Cross-provider grant translation is not a protocol (§6).
- OQ-82: every Object below the Manifest, including indexes/codebooks/graphs, is encrypted (§2).
- OQ-83: v1 pins AES-256-SIV; another suite requires a new critical tracks form, not an ignorable algorithm value.
- OQ-84: GC requires the key for encrypted reference-bearing Objects and fails closed (§6).
- OQ-85: quota counts ciphertext envelope bytes (§6).
- OQ-86: the mandatory vectors are now defined and shipped with this revision (§9).
- OQ-87: v1 has no per-modality opt-out (§6).
Searchable encryption, homomorphic vector search, PIR, token-level encryption, encrypted Refs/Genesis/Manifests, per-user encryption and per-Item crypto-shredding are out of scope. Backend-managed SSE remains a useful deployment control but is transparent to, and not an implementation of, this protocol format.