# Semantic search

End-to-end: take a natural-language query (`"a dog playing in grass"`), encode it to a CLIP-512 vector in the browser, run `Dataset.iterVector`, render the top-K. No server inference — the embedding model runs in the user's browser via `@xenova/transformers`.

## Prerequisites

The dataset must have been ingested with **CLIP-512 image embeddings** (`embedding.f32.dim=512.bucketed` modality). The Python tool that bakes this:

```bash
python ingest_imagenet100_clip.py --ivf --rabitq --rerank
```

(see `vortex/vortex-dataset-python/examples/` for the live script). The image encoder and the text encoder share a vector space — text queries land in the same coordinate system as image embeddings.

## Install the text encoder

```bash
pnpm add @xenova/transformers
```

`@xenova/transformers` is the browser-friendly port of Hugging Face transformers. It downloads the CLIP text encoder model the first time it's used (~150 MB FP32; ~38 MB if you can use `quantized: true`) and caches it in IndexedDB.

## Encode + query

```ts

// Lazy-load the text encoder; reuse across queries.
let extractor: any = null;
async function ensureExtractor() {
  if (!extractor) {
    extractor = await pipeline(
      "feature-extraction",
      "Xenova/clip-vit-base-patch32",
      { quantized: false }, // full precision for model-parity with image-side
    );
  }
  return extractor;
}

async function semanticSearch(
  ds: Dataset,
  text: string,
  topK = 12,
): Promise<{ anchor: bigint; score: number; label?: string }[]> {
  const e = await ensureExtractor();
  const out = await e(text, { pooling: "mean", normalize: true });
  const vec = Array.from(out.data) as number[];

  const batches = await ds.iterVector({
    field: "embedding",
    query: vec,
    topK,
  });

  const results: { anchor: bigint; score: number; label?: string }[] = [];
  for (const batch of batches) {
    for (let i = 0; i < batch._time_anchors.length; i++) {
      results.push({
        anchor: batch._time_anchors[i],
        score: (batch._scores as Float32Array)[i],
        label: (batch.label as string[] | undefined)?.[i],
      });
    }
  }
  return results;
}
```

## React UI

```tsx
function SearchBar({ ds }: { ds: Dataset }) {
  const [q, setQ] = useState("");
  const [hits, setHits] = useState<Awaited<ReturnType<typeof semanticSearch>>>([]);
  const [busy, setBusy] = useState(false);

  const onSearch = async () => {
    setBusy(true);
    setHits(await semanticSearch(ds, q, 24));
    setBusy(false);
  };

  return (
    <div>
      <input
        value={q}
        onChange={(e) => setQ(e.target.value)}
        onKeyDown={(e) => e.key === "Enter" && onSearch()}
        placeholder='e.g. "close-up of a hand applying eyeshadow"'
      />
      <button onClick={onSearch} disabled={busy || !q}>
        {busy ? "encoding…" : "search"}
      </button>
      
    </div>
  );
}
```

## First-search latency

| Step | Cost |
|---|---|
| Model download (first time, cached after) | ~5–15 s (cold network) / instant on subsequent visits |
| Text encoding | 30–100 ms |
| `iterVector` (server-side ADC + rerank) | 200–500 ms depending on probe-count |
| Thumbnail fetch | 1 HTTP GET per pack (packed Fragments) |

After the first query the model is hot; subsequent searches are sub-second end-to-end.

## Filter by label

`iterVector` accepts `whereEq`. Restrict to a specific class:

```ts
const dogs = await ds.iterVector({
  field: "embedding",
  query: textVec,
  topK: 5,
  whereEq: { label: "golden retriever" },
});
```

When `whereEq` is set, the filtered scalar fields are auto-populated on each result batch.

## Model parity caveat

`@xenova/transformers` ships several CLIP variants. To match the image-side embeddings exactly, use the **same checkpoint** that produced the dataset (typically `Xenova/clip-vit-base-patch32` for the public ImageNet fixture). With `quantized: false` the text encoder is bit-comparable to the Python `open_clip` ViT-B/32; with quantization on you lose ~1% top-K recall — usually acceptable.