Semantic search

End-to-end: take a natural-language query ("a dog playing in grass"), encode it to a CLIP-512 vector in the browser, run Dataset.iterVector, render the top-K. No server inference — the embedding model runs in the user's browser via @xenova/transformers.

Prerequisites

The dataset must have been ingested with CLIP-512 image embeddings (embedding.f32.dim=512.bucketed modality). The Python tool that bakes this:

bash

python ingest_imagenet100_clip.py --ivf --rabitq --rerank

(see vortex/vortex-dataset-python/examples/ for the live script). The image encoder and the text encoder share a vector space — text queries land in the same coordinate system as image embeddings.

Install the text encoder

bash

pnpm add @xenova/transformers

@xenova/transformers is the browser-friendly port of Hugging Face transformers. It downloads the CLIP text encoder model the first time it's used (~150 MB FP32; ~38 MB if you can use quantized: true) and caches it in IndexedDB.

Encode + query

import { pipeline } from "@xenova/transformers";
import { Space, Dataset, S3Backend } from "@dreamlake/dreamdb";

// Lazy-load the text encoder; reuse across queries.
let extractor: any = null;
async function ensureExtractor() {
  if (!extractor) {
    extractor = await pipeline(
      "feature-extraction",
      "Xenova/clip-vit-base-patch32",
      { quantized: false }, // full precision for model-parity with image-side
    );
  }
  return extractor;
}

async function semanticSearch(
  ds: Dataset,
  text: string,
  topK = 12,
): Promise<{ anchor: bigint; score: number; label?: string }[]> {
  const e = await ensureExtractor();
  const out = await e(text, { pooling: "mean", normalize: true });
  const vec = Array.from(out.data) as number[];

  const batches = await ds.iterVector({
    field: "embedding",
    query: vec,
    topK,
  });

  const results: { anchor: bigint; score: number; label?: string }[] = [];
  for (const batch of batches) {
    for (let i = 0; i < batch._time_anchors.length; i++) {
      results.push({
        anchor: batch._time_anchors[i],
        score: (batch._scores as Float32Array)[i],
        label: (batch.label as string[] | undefined)?.[i],
      });
    }
  }
  return results;
}

React UI

tsx

function SearchBar({ ds }: { ds: Dataset }) {
  const [q, setQ] = useState("");
  const [hits, setHits] = useState<Awaited<ReturnType<typeof semanticSearch>>>([]);
  const [busy, setBusy] = useState(false);

  const onSearch = async () => {
    setBusy(true);
    setHits(await semanticSearch(ds, q, 24));
    setBusy(false);
  };

  return (
    <div>
      <input
        value={q}
        onChange={(e) => setQ(e.target.value)}
        onKeyDown={(e) => e.key === "Enter" && onSearch()}
        placeholder='e.g. "close-up of a hand applying eyeshadow"'
      />
      <button onClick={onSearch} disabled={busy || !q}>
        {busy ? "encoding…" : "search"}
      </button>
      <SearchResults hits={hits} />
    </div>
  );
}

First-search latency

Step	Cost
Model download (first time, cached after)	~5–15 s (cold network) / instant on subsequent visits
Text encoding	30–100 ms
`iterVector` (server-side ADC + rerank)	200–500 ms depending on probe-count
Thumbnail fetch	1 HTTP GET per pack (packed Fragments)

After the first query the model is hot; subsequent searches are sub-second end-to-end.

Filter by label

iterVector accepts whereEq. Restrict to a specific class:

const dogs = await ds.iterVector({
  field: "embedding",
  query: textVec,
  topK: 5,
  whereEq: { label: "golden retriever" },
});

When whereEq is set, the filtered scalar fields are auto-populated on each result batch.

Model parity caveat

@xenova/transformers ships several CLIP variants. To match the image-side embeddings exactly, use the same checkpoint that produced the dataset (typically Xenova/clip-vit-base-patch32 for the public ImageNet fixture). With quantized: false the text encoder is bit-comparable to the Python open_clip ViT-B/32; with quantization on you lose ~1% top-K recall — usually acceptable.