Spec 0018 — Multi-Tenant Operation

Status: Draft (Phase 4 design). The enforcement primitives below — capability-token validation, quota responses, fair-share scheduling, TenantUsageBatch emission — define the contract for a DreamDB-aware mediation gateway (0000 §3), not for the commodity object store of spec/0005. No reference gateway ships in v0; this is a contract for a v0.X component. 0019 likewise has no reference encryption implementation, although its wire format is now normative. The isolation primitives (§2 — distinct Genesis/Timeline IDs + one-bucket-per-Space + native auth) work on plain object storage today. See §1.1 for the split. Depends on: spec/0001, spec/0005, spec/0006, spec/0008, spec/0012. Motivation: A single-tenant DreamDB deployment works because the operator implicitly owns all storage budgets, query capacity, and access control. Any production SaaS hosting multiple customers — or any internal platform serving multiple teams — needs explicit primitives: per-tenant resource isolation (quota + rate limits), per-tenant identity (so a token from tenant A can't read tenant B), per-tenant cost accounting (so the billing layer knows who consumed what), and noisy-neighbor protection (so one tenant's pathological query can't starve every other tenant). Without these primitives, every multi-tenant deployment reinvents them inconsistently. spec/0018 pins the contract.

1. Purpose

The Space concept (0001 §8) is a possible deployment tenant boundary, not a cryptographic access-control mechanism. Distinct Timeline identities separate names; backend authorization separates access. Operational isolation is another obligation: one tenant's scan must not monopolize another's capacity under the gateway policy proposed in §1.1.

By the end of this document the following are concrete:

• The Space-config quota fields: per-Space storage cap, queries/sec, concurrent streams, GET/PUT bandwidth.
• The verified authorization context required at the gateway (§3), independently of the deployment's credential encoding.
• The TenantUsageBatch ObjectKind: per-Space rolling usage statistics published by the gateway; the cost-accounting source of truth.
• The rate-limit response contract: when quota is exceeded, the gateway returns HTTP 429 with Retry-After; SDKs handle this without conflating with retryable transient failures.
• The fair-share scheduling discipline: when multiple tenants share a single gateway, query and ingest cost MUST be distributed fairly (no single tenant captures >50% steady-state by default).
• The cross-tenant federation contract: each backend authorizes its own request independently; federation carries neither credentials nor implicit privilege.

What stays defined elsewhere:

• Per-Space cryptographic identity (Genesis Object) — spec/0001.
• HTTP semantics — spec/0005.
• Federation membership and runtime Connector bindings — spec/0012; these do not define or convey credentials.

What this document does NOT define:

• Billing. Cost accounting produces TenantUsageBatch Objects; how operators convert them to invoices is operator-layer.
• Authentication / identity provisioning and credential wire formats. Operators select a complete authentication scheme and trusted issuers (§3). This draft does not standardize a DreamDB token, signature suite or key-discovery service (OQ-78).
• Quota enforcement consistency model. Best-effort eventually-consistent quota tracking is the v0.X contract; strict transaction-bounded quota requires consensus and is out.
• Per-modality quotas. Storage budget is per-Space; per-modality sub-allocation is operator-layer.

1.1 Where enforcement runs (the trust boundary)

Multi-tenant operation splits into two halves with very different deployment requirements, and conflating them is a category error:

Namespace separation plus backend authorization. Different Genesis Objects produce distinct content-addressed Timeline identities, but knowledge of a hash is not an access-control boundary. Separate buckets and correctly scoped backend credentials can isolate tenants without a DreamDB-aware server. That isolation comes from authorization, not from the non-collision property of Timeline IDs; payload confidentiality is a separate encryption concern (0019).

Enforced policy — requires a mediation gateway; not implementable on plain S3/MinIO. Capability-token verification (§3), quota enforcement (§2.2), fair-share scheduling (§5), and TenantUsageBatch emission (§4) are all policy decisions evaluated against a potentially-adversarial client. By a basic security property they cannot live in the client's own SDK (the adversary controls it), and a commodity object store cannot evaluate a DreamDB-defined policy — it cannot parse a Manifest's Space-config, verify an Ed25519 capability token, or schedule by tenant_id. These primitives therefore presuppose a trusted, DreamDB-aware mediation gateway interposed on the HTTP boundary (0000 §3, "the optional mediation gateway"): it implements the spec/0005 HTTP contract (so it is transparent to the SDK), enforces per-tenant policy, then forwards to the object store with privileged credentials.

Throughout this document, "the gateway" denotes that trusted enforcement point. Where the text says the gateway MUST verify a token or return 429/507, it means this layer — not the commodity object store of spec/0005. A deployment running only plain object storage gets the isolation half and none of the enforcement half.

2. The Space as the tenant boundary

A Space (spec/0001 §1) is one writer's universe — one Genesis Object, one or more Timelines, one Ref namespace. Operationally a Space is owned by exactly one tenant, identified by a stable tenant_id.

tenant_id is OPAQUE to the protocol — a UTF-8 string up to 256 bytes. Operators choose its format (UUID, email, account-id; example: acme-corp-prod). The protocol cares only that it's stable per Space and verifiable in capability tokens.

2.1 Space-config quota fields

The Manifest's Space-config sub-Object (spec/0002 §7.2.0) gains an OPTIONAL quotas field:

"space_config": {
  …,
  "quotas": {
    "tenant_id":          "<utf-8 string>",
    "max_storage_bytes":  <u64 | null>,                  ;; null ⇒ unbounded
    "max_queries_per_sec": <u32 | null>,
    "max_writes_per_sec":  <u32 | null>,
    "max_concurrent_streams": <u32 | null>,
    "max_get_bytes_per_sec": <u64 | null>,
    "max_put_bytes_per_sec": <u64 | null>,
    "max_concurrent_reencodes": <u32 | null>,            ;; spec/0017 jobs
  }
}

Absent quotas ⇒ unbounded (single-tenant deployment behavior).

The quotas live in the Space-config because they're part of the Space's identity from the gateway's perspective (it reads Space-config to know each tenant's limits). An operator changing a tenant's quota publishes a new Manifest with updated Space-config and updates refs/main via CAS — same machinery as any other registry change.

2.2 Enforcement contract

When a request would exceed a quota:

• Storage cap reached → PUT returns HTTP 507 Insufficient Storage. The gateway includes a DreamDB-Quota: storage,used=<N>,limit=<M> header.
• Rate limit reached → GET / PUT returns HTTP 429 Too Many Requests with Retry-After: <seconds>. Same DreamDB-Quota header naming the offending resource.

SDKs MUST distinguish 429 (quota) from 5xx (transient backend failure):

• 429 → surface as QuotaExceeded error; the caller decides whether to wait and retry.
• 5xx → spec/0005 exponential-backoff retry path; the SDK handles transparently.

2.3 Quota measurement granularity

Token-bucket per (tenant_id, resource) is the recommended gateway implementation. Bucket refill rate equals the quota; bucket depth allows brief bursts.

• Refill rate: per the quota.
• Bucket depth: typically 5–10× the per-second rate (allows 5–10 second bursts).
• Reset on rate-window boundaries; no carryover.

This is implementation-defined; the wire contract is just "exceed → 429 + Retry-After."

3. Gateway authorization boundary; no portable token format

Authentication is deployment-local. This draft specifies the policy applied after successful authentication, not bytes to sign or a token to exchange. The earlier CBOR token sketch is withdrawn: it supplied neither a complete signature contract nor a trust bootstrap and MUST NOT be treated as a standard DreamDB credential. Current 0012 supplies no missing token or discovery rules.

The deployment's authentication adapter supplies a verified principal, tenant identity, authorized operations and address scope. These are semantic inputs to policy, not new Manifest fields or a CBOR schema. They MUST come from trusted server configuration and a verified credential/session; copying client-provided tenant or scope fields into this context does not authenticate them. A profile using signed tokens must specify its signed bytes, suite, issuer/key trust, validity/time units and request credential transport before deployment. Missing or unsupported authentication profiles fail closed; federation membership is never an issuer trust source. No interoperable DreamDB token is claimed here.

OQ-78 retains the separate decision whether to standardize such a portable token profile; OQ-92 retains the gateway implementation. Neither is closed by defining this boundary, and no stored credential format is migrated by this revision.

3.1 Verification at the gateway

For every request, the gateway MUST:

1. Authenticate under the selected deployment profile, including signature and validity checks where that profile uses signed credentials.
2. Resolve the target Space and tenant using operator-controlled configuration, not an untrusted request's claimed owner or a tenant-editable Manifest alone.
3. Require the verified tenant identity to match that target tenant.
4. Require the verified authorization to cover both the operation and resolved address, using the deployment's path interpretation. A matching path without operation permission (for example, read permission on a PUT) is insufficient.

An authenticated request under tenant bob targeting a Space owned by alice, or an authenticated request lacking operation/path permission, MUST be rejected with HTTP 403 Forbidden before forwarding. Authentication failure follows the chosen authentication profile and never enters the authorized path.

3.2 Cross-tenant federation safety

Federation (0012) creates no cross-tenant credential and no implicit trust hop. Each runtime Connector authenticates independently to the coordinator or child backend it addresses, and every gateway applies §3.1 to that request's own tenant and path. The caller supplying backend-id-to-Connector bindings is a trusted deployment component; binding an id never widens a token's tenant or path scope.

An operator copying a child closure between tenants performs ordinary GET and PUT operations authorized separately at their source and destination. A token accepted by one backend is not thereby accepted by another, and federation never preserves, exchanges, or elevates a token across a hop.

4. The TenantUsageBatch Object

Cost accounting needs a protocol-level signal. spec/0018 defines a gateway-emitted ObjectKind that summarizes per-Space resource consumption over a rolling window.

4.1 Address path

tenant-usage/<batch-hash>

(New top-level namespace, parallel to manifests/, refs/. The bucket is operator-defined — typically one bucket per backend, but a federation MAY have one per region.)

4.2 CBOR encoding

{
  "version":         1,
  "tenant_id":       "<utf-8 string>",
  "window_start":    <u64>,                    ;; Unix ns
  "window_end":      <u64>,                    ;; Unix ns
  "metrics": {
    "storage_bytes_avg":    <u64>,             ;; average over window
    "storage_bytes_peak":   <u64>,
    "get_count":            <u64>,
    "get_bytes":            <u64>,
    "put_count":            <u64>,
    "put_bytes":            <u64>,
    "list_count":           <u64>,
    "delete_count":         <u64>,
    "query_count":          <u64>,             ;; SDK-level Query/HybridQuery invocations
    "stream_seconds":       <u64>,             ;; aggregate concurrent-stream time
    "reencode_items":       <u64>,             ;; spec/0017 progress
  },
  "violations": [                              ;; OPTIONAL: 429 / 507 events
    { "code": "rate_limit_exceeded",
      "resource": "max_queries_per_sec",
      "count": <u32>,
      "first_at": <u64>,
      "last_at": <u64>
    },
    …
  ],
  "previous_batch":  <multihash | null>,       ;; chain for continuous billing series
}

4.3 Publication cadence

Gateways emit TenantUsageBatch Objects on a schedule (recommended: every 1 hour or every 1 GB of activity, whichever first). The chain (previous_batch link) gives operators a content-addressed audit trail.

The most recent TenantUsageBatch per Space is exposed via a Ref:

tenant-usage-refs/<tenant_id>

Pointing at the latest batch hash. Operators poll this Ref for billing extracts.

4.4 Trust model

TenantUsageBatch is produced by the gateway operator. Tenants MAY verify their own data by sampling (each batch references its predecessor by hash — a tampering operator would need to rewrite the entire chain). For production billing trust, operators are expected to log batches to an independent audit system; cryptographic verifiability of every counter requires per-operation signatures (out of scope for v0.X).

5. Fair-share scheduling

When a single gateway fronts multiple tenants, the gateway's scheduler MUST distribute capacity fairly. v0.X requires:

5.1 Per-tenant queues

The gateway maintains a queue per tenant (or per (tenant, resource) pair). Requests are admitted from queues in a fair-share order — typically Weighted Fair Queueing or Deficit Round Robin.

5.2 Anti-monopoly invariant

In steady state, no single tenant SHOULD capture more than 1 / N_active_tenants of total capacity for any single resource, where N_active_tenants is the count of tenants with non-empty queues. This bounds noisy-neighbor impact.

(Exception: if other tenants are idle, a single active tenant MAY use up to its full quota. The "more than" bound applies only when contention exists.)

5.3 Pathological-query protection

A single DreamDB query that touches many Bucket Objects (e.g. a federated 100-shard scatter-gather) can consume disproportionate gateway/backend capacity. The gateway MAY:

• Limit fan-out concurrency per tenant (recommended default: 16 concurrent in-flight requests per tenant).
• Throttle large-byte responses progressively as the tenant's bucket depletes.
• Surface query-cost-estimate headers (DreamDB-Query-Cost: <estimated-units>) on responses, letting tenants self-throttle.

These are implementation-defined; the contract is just "no single query starves the rest."

6. Tenant onboarding and offboarding

6.1 Onboarding

1. Operator enrolls the tenant under the deployment's authentication profile; a tenant signing key is required only if that profile requires one.
2. Operator publishes a Manifest with Space-config carrying:
   - tenant_id: <new-tenant-id>
   - quotas: <starting quotas>
3. Operator provisions credentials under the deployment profile, authorizing the tenant's write operations on this Space.
4. Tenant SDK uses those credentials through its Connector to start writing.

6.2 Offboarding

1. Operator publishes a Manifest with Space-config quotas all set to 0.
   → tenant can no longer write or query.
2. Optional: operator schedules a snapshot-rollup (spec/0008 §9.3) to consolidate
   the tenant's data into a final archival Manifest.
3. After a retention window (operator-defined; typically 30-90 days):
   - All tenant Refs are deleted.
   - GC reclaims all reachable Objects.
4. Final TenantUsageBatch is published for billing closure.

Offboarding is a clean operation in DreamDB because content-addressing makes the storage trivially GC-able once Refs are gone. No "delete user data" scan-and-purge across tables.

6.3 GDPR / right-to-be-forgotten

A user-level "delete my data" request requires the operator to:

1. Identify which Items in the tenant's Space contain the user's data.
2. Publish a Layer Track with a tombstone marking those Items.
3. After the tombstone propagates and any active sessions reach the new Manifest, schedule a snapshot-rollup that physically excludes the tombstoned Items from the new Manifest's index.
4. GC reclaims the excluded Items after the safety threshold.

This is operator-driven; the protocol provides the primitives. Per spec/0008, immutability + Layer composition gives the right semantics.

7. Conformance categories (per spec/0009 §8.6.4)

8. Sizing and operational notes

8.1 Quota granularity

Default quotas for a "typical" tier:

These are operator suggestions, not protocol-mandated. The protocol specifies the wire contract; the values are deployment-policy.

8.2 TenantUsageBatch storage cost

For 1000 tenants, 1-hour cadence, 90-day retention:

• 24 × 90 = 2160 batches per tenant.
• ~2 KB per batch (~500 metric ints + small overhead).
• Total: 1000 × 2160 × 2 KB = ~4.3 GB.

Negligible at production scale.

8.3 Latency overhead

Quota checks add ~1 ms per request (in-memory token-bucket check). Acceptable; comparable to per-request auth.

9. Out of scope

• Cross-region tenant quotas. Each federation participant tracks its own per-tenant quota; a tenant exceeding global quota by spreading load across regions is an operator-policy concern.
• Predictive quota. "Tenant will hit cap in 3 days" warnings — operator-layer analytics.
• Resource bin-packing. Which backend a new tenant lands on — operator-layer placement.
• Credential revocation. The selected authentication profile defines expiry and revocation. This draft neither mandates short-lived tokens nor prohibits revocation lists.

10. Open questions

• OQ-75 (→ this spec): Authentication and non-equivocation of TenantUsageBatch history. Hash links prove consistency only relative to a trusted root. A signature authenticates its signer but does not alone stop that signer rewriting or equivocating; the proposal needs an independently retained checkpoint/trust policy before claiming history cannot be rewritten.
• OQ-76 (→ this spec): Multi-region quotas. If a tenant has 100 GB across 3 federated backends, is their effective quota 100 GB total or 100 GB per backend? Probably per-backend for operational simplicity; aggregation is a billing-layer concern.
• OQ-77 (→ spec/0009): Conformance vectors for fair-share scheduling — synthetic load over N tenants asserting the anti-monopoly invariant. Block multi-tenant conformance on this.
• OQ-78 (→ this spec): Decide and define a portable gateway token profile, if needed, including tenant scope, signed bytes, suite, validity units and issuer trust. §3 currently defines only the deployment-local authentication-to-policy boundary; federation does not carry authorization.
• OQ-92 (→ this spec; formerly ledger-only): Reference mediation gateway and capability-scoped conformance for the enforcement boundary described in 0000 §3. Ordinary object storage and a browser presigning adapter do not enforce this draft's quotas, tenant authorization or scheduling policy.

Next: spec/0019 — the normative data-plane encryption format. A reference writer and reader remain unimplemented. Last spec in the Phase-4 batch.