Changelog

Native Rust SQL v1 — 2026-09-15 (source only)

• #377 / PR #378: opt-in dreamdb-sql, with a closed read-only SELECT/WHERE/LIMIT subset, typed parameters, pinned sessions, EXPLAIN and typed column batches.
• Reuses scalar indexes and vector/BM25 search, with late projection, explicit scan opt-in and query-local resource limits. Search WHERE is post-top-k; eager internals are not streaming or hard-memory-bounded.
• No storage-format change, SQL write path, Python/JS/WASM adapter, CLI SQL command or package release. See usage and performance boundaries.

Python 0.0.12 / npm 0.5.4 — 2026-09-10

• #365: authenticate Bucket metadata and retain reference-record VectorStorage dependencies during GC/copy traversal, for inline and paged indexes.
• #342: query-local rerank pool and exactness policy, including Python/WASM.
• #343–345: Rust fragmentation observations, explicit maintenance plans and bounded Fragment repacking.
• #346: independently bound IVF/global-vector PQ composition evidence.
• #356–360: Rust Reencode admission, Ref Watch, snapshot tags, verified copy plans and media Stream selection; no automatic SDK wrapper or daemon claim.
• Synchronize the site's spec corpus with core a119479: 28 documents, 77 resolved / 27 open OQs, 16 Draft headers. These are design dispositions, not an implementation-completeness percentage.

See release notes for GC upgrade and capability boundaries.

Python 0.0.11 / npm 0.5.3 — 2026-09-10

Both packages are available from their registries. The release groups the following changes; see release notes for upgrade cautions.

• #283/#295: capacity-aware bucket slicing, aligned exact sidecars, per-run rewrite quotas, shared CLI/Dataset compaction, Schema/SDK payload-cap configuration and bounded per-cell merge work.
• #270–276: backfill claims, canonical multi-vector records, persistent scalar B-trees, paged tombstones, federation, rendition playlists and reference compression.
• #316–328: progressive geometry, runtime embedding identity checks, raw Fresh graphs, graph routing and compressed graph search, L2, entity keys, ragged/CSR, field-qualified and typed HotShards, scalar path-copy updates and hybrid planning.
• #278 and spec reconciliation: contract changes are documented separately from implemented SDK capabilities; no general encryption implementation is claimed.

Historical documentation snapshot — 2026-09-10

At that snapshot, the site reflected core main 65c8e05: 28 specs, 66 resolved / 38 open OQs, and 16 Draft headers. Newly documented implementations include optional entity keys, progressive geometry, structured ragged/CSR, L2, compressed/routed/Fresh graphs, typed HotShards, embedding identity checks, scalar path-copy updates and hybrid planning. See the feature guide and limits.

That documentation publication was not an SDK package release. At the time, Python 0.0.10 and npm 0.5.2 were the published versions and these examples required a source build. This paragraph records that historical state, not current registry availability.

All notable changes to DreamDB are documented here. Reference implementation status; protocol-spec changes are noted with their spec doc reference. Dates use ISO format.

[Unreleased]

Protocol and conformance sync (2026-09-08)

• Synchronized the generated Specification section with dreamdb-core through merge adeea46, including the work tracked in issues #270–#291 and #295.
• Added the settled wire contracts and conformance material for lineage-v2 backfill claims, canonical multi-vector records, value-keyed scalar B-trees, paged Timeline-scoped tombstones, FederationManifest, adaptive-bitrate rendition playlists, reference-mode vector compression, and post-v1 dense/ragged/sparse typed arrays.
• Defined Lineage-v3 encrypted Object envelopes, embedding realization identity, critical-extension deployment rules, Vamana entry-point behavior, federation guarantees, range-read validation, and schema-migration coverage semantics.
• Updated compaction to describe capacity-bounded F → K bucket consolidation, exact-sidecar alignment and bounded per-cell memory; clarified GC root-set and age-threshold safety.
• Corrected the specification index and all generated spec pages to their current normative status and open-question counts.

Added — AWS S3 / SigV4 production path (2026-05-22)

• dreamdb-connector-http/src/sigv4.rs — S3Signer wraps the aws-sigv4 crate; produces Authorization + x-amz-date + x-amz-content-sha256 + (optionally) x-amz-security-token headers per request.
• HttpConnectorConfig.signer_from_env() auto-detects AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN / AWS_REGION (falls back to us-east-1). No signer attached → connector sends unsigned requests (MinIO-anonymous dev mode preserved).
• Every connector verb (PUT / GET / HEAD / LIST / DELETE / multi-range GET) now signs when a signer is configured; SigV4 covers all wire headers including per-request If-Match, Range, etc.
• dreamdb-cli/src/main.rs and dreamdb-dataset-python/src/lib.rs factories pick up the env-var signer automatically — same code talks to MinIO and S3.
• Verified live against s3://dreamdb-test-20260518/ in us-east-1: create + append + count + snapshot + branch + history + delete all round-trip; 69 Objects landed, 19.8 KB total.

Added — Python SDK ergonomics (2026-05-22)

• Dataset.iter_stream(batch_size, fields, channel_buffer) (B1.5): real Python generator backed by a tokio mpsc channel. Bounded RAM per batch (verified 104 MB peak RSS on 91K records). Closes the last 10B-scale follow-up.
• Dataset.delete(anchors, reason=None) — tombstone anchors via spec/0020. Returns the new Manifest hash.
• Dataset.tombstone_set() — resolved set of suppressed anchors at the current Manifest.
• Dataset.merge(other_ref, strategy="fast-forward"|"union-tracks") — full Python parity with the Rust MergeStrategy enum.
• Dataset.merge_many(branches) — N-way sequential union-merge for sharded ingest.
• Dataset.history(max_depth=50) — walks the Manifest DAG via parents[0]; returns [{manifest, ts_ns, writer, parents_count, tracks_count}, …]. Mirrors the browser UI's ⏳ history button.
• Dataset.list_refs() — lex-sorted list of every ref under the backend's refs/ prefix.
• Dataset.count() + __len__ — record count at the current tip, via iter_stream. First-time-user reflex now works.
• Schema is chainable: (vd.Schema().add_image(...).add_embedding(...).add_scalar_categorical(...)). Wrapped _RustSchema; backward compatible.
• Friendlier HTTP errors: 403 / 404 / 412 / connection-refused / SI-conflict / schema-mismatch errors now carry actionable hints (e.g. "for local MinIO dev: mc anonymous set public local/<bucket>").

Added — CLI ergonomics (2026-05-22)

• dreamdb query --backend ... --ref-name ... --field <name> --query-file <path> — top-K vector search from the command line. Operator spot-check verb; reads raw LE-f32 query bytes from a file.
• dreamdb delete --ref-name ... <anchor> [<anchor>...] — tombstone CLI (paired with the SDK method).
• dreamdb merge-many --ref-name <trunk> <branches>... — orchestrator CLI for sharded-ingest workflows.
• Fixed: dreamdb snapshot --help no longer shows the GC description (doc-comment block at main.rs:207 was attached to the wrong variant); dreamdb gc --help now has its own help text.

Added — operator manifests (2026-05-22)

• dreamdb-cli/examples/sharded-ingest.yaml (228 lines) — 3-Job k8s pattern for N parallel workers + merge-many orchestrator.
• dreamdb-cli/examples/ada-ivf-step-sharded.yaml (246 lines) — 4-stage pipeline (centroids → publish-SI → redispatch → finalize) for B3's sharded redispatch at 10B-scale.
• dreamdb-cli/examples/ada-ivf-step.yaml — header updated with "when to use which YAML" pointer.

Added — documentation (2026-05-22)

• README.md rewritten (40 → 165 lines): pitch + 4 first principles + Python sketch + status table + 60-second quickstart + repo layout + spec roadmap with honest gaps. Now reflects the actual shipped state.
• docs/tutorial.md (370 lines): "DreamDB in 10 minutes" end-to-end walkthrough — schema + ingest + snapshot + query + PyTorch DataLoader + time-travel + ada-ivf-status + delete + sharded ingest. Step 14 documents the S3 migration path.
• INDEX.md: refreshed to 21 specs (was 20); added spec/0020 row; "What's Next" updated to reflect shipped state.

Spec changes (2026-05-18 → 2026-05-22)

• spec/0001 §2.1 — Object-kinds table expanded from 10 to 17 entries (added TombstoneList, ScalarIndex, VectorCompressor, ItemManifest, GraphIndex, GraphPage, plus existing ones that were missing).
• spec/0008 §5.3 (new) — formalizes the distinction between layered-merge (the original §6.1 approach: both parents' Tracks coexist as separate TrackEntrys) and fused-merge (B2's implementation: one merged TrackObject per modality with cell-by-cell bucket reconciliation). Reference implementation ships fused-merge for SpatialBucket tracks; both are spec-valid.
• spec/0020 §3.1 — broken cross-reference "spec/0001 §3.2" corrected to "spec/0002 §7.5" (path table actually lives in spec/0002).
• spec/0020 §3.1 — TombstoneEntry anchor field clarified to be u64 (the Item-level TimeAnchor), not Multihash. Single anchor suppresses every record across every modality.

Changed — dreamdb-dataset/src/dataset.rs split into 10 modules

The monolithic 5592-line file was split into a slim facade (dataset.rs, 184 lines) plus 9 single-responsibility submodules:

Test suite unaffected: 721 tests still pass after the split.

Added — 10B-scale blocker push (2026-05-15 → 2026-05-18)

All 7 execution blockers from design/0006-10b-scale-blockers.md shipped; B7 audit verified no-op:

Latent bug discovered en route: post-FastForward field_tracks wasn't being refreshed → reads returned stale pre-merge results. Fixed via Dataset::refresh_field_tracks_from_current().

Test suite: 705 → 721 tests, 0 regressions.

Other quality work

• Spec audit: 6 stale-claim fixes across design/0001-0007 (Phase-0 banners refreshed, "Still no tombstones" → "✅ shipped", phantom spec/0021 references removed, etc.).
• Unused-import warnings cleanup: cargo fix + manual sweep removed ~30 dead imports across dreamdb-dataset, dreamdb-protocol, dreamdb-bench.

Live evidence

• 231K imagenet-100 ingested with CLIP embeddings, RaBitQ corrected, IVF k=70. Validated linear-probe training (val_acc 4.9% → 36.0% in 5 epochs).
• B1.5 streaming verified mid-ingest on a half-baked 91K-record bucket: 5400 records/sec, 104 MB peak RSS, 425ms first-batch latency.
• AWS S3 SigV4 verified on s3://dreamdb-test-20260518/ (us-east-1): full Dataset lifecycle round-trips (create/append/count/snapshot/branch/history/delete) in 32.6s for 50 samples (WAN-bound, not DreamDB-bound).
• 1.33M imagenet-1k-256 ingest aborted at ~162K records due to local disk pressure; the experiment validated SDK + B2/B3 mechanics; the partial bucket was reclaimed during disk cleanup.

Pre-history

This changelog starts with the 10B-scale push. Earlier work (Phase 1-3.4: dataset platform, schema persistence, IVF+RaBitQ foundation, Ada-IVF maintenance, paged tracks, chain-aware lineage, browser SDK) is documented in design/0002-known-flaws-retrospective.md and the per-spec status fields. The protocol spec (spec/0000-0020) has been stable since 2026-05-13.