Why a gate
The problem was never the transform. Every embedding migration has two parts: computing a mapping between two embedding spaces, and deciding whether that mapping is good enough to ship. The first is a solved regression problem. The second is the part that decides whether your migration from re-embedding actually holds up — and it is the part most tooling skips.
Computing a mapping between two embedding spaces is easy. Embed the same ~2K calibration texts with both models, fit a ridge regression, and you have a transform. On BEIR SciFact, Isotrieve's default ridge mapping retains 86–92% of retrieval quality versus full re-embedding. Any library can fit that matrix.
The risk is not in the matrix — it is in retention: whether the mapped vectors still retrieve the right neighbors on your corpus, your queries, your tolerance for silent degradation. That is a validation problem, and no transform, however clever, answers it for you.
Failure modes a transform can't see
These are exactly the failures that are invisible at fit time but show up in production. A mapping library reports a fitted loss; none of these show up in that number.
- Hubness — mapped vectors can crowd around dense regions, so a handful of documents become near-neighbors to almost every query. The map fits a global regression; it cannot see per-vector crowding.
- Seed sensitivity — different calibration seeds preserve different neighborhoods. One seed can look clean and another degrades, so a single fitted run is not evidence of quality.
- Domain mismatch — pretrained adapters are fit on general data. On your distribution, the same transform can silently lose relevance with no error raised.
Isotrieve is Migration CI: it measures retention on held-out source/target samples and returns a verdict — PASS, WARN, or FAIL — so you decide to ship based on data, not hope. If you are planning to migrate an embedding model, this is the part that turns the project from a gamble into a reviewed change.
Adapter library vs Isotrive
Honestly: adapter libraries are good at what they do. They translate between embedding spaces using pretrained adapters for popular model pairs. They solve a different problem.
| Adapter libraries | Isotrive | |
|---|---|---|
| Mapping source | Pretrained on general data | Fit on your corpus |
| Where vectors live | Translated at query time | Migrated in place, in your store |
| Failure mode | You find out in production | Gate fails the build |
| Domain corpora | General-purpose fit | Calibrated to your distribution |
Use an adapter library when you want to query an existing index with a different model today — a small store, a short-lived experiment, an A/B test, or a prototype that may never go to production. Query-time translation is the right tool when the work is transient.
Use Isotrive when the migration is permanent: you want to move the index itself into the new space, in place, and be able to prove it did not degrade. That is the gap the gate fills — adapter libraries hand you a transform and wish you luck; Isotrive tells you whether to ship it.
Bring your own transform
"Bring your own transform — the gate decides." You are not locked into an Isotrive-fitted map. If you already have a transform from another tool, the gate still works on it.
ExternalMapping wraps any callable (np.ndarray) -> np.ndarray — including pretrained adapters from libraries like EmbeddingAdapters — so you can gate a transform you never fitted yourself. Isotrieve's value is the verdict, not the matrix.
We are equally honest about where the technique comes from. Isotrieve builds directly on prior work:
- vec2vec (arXiv:2505.12540) — embedding-space translation via linear maps
- mini-vec2vec — the lightweight, practical follow-up
- Drift-Adapter — validates Procrustes and MLP-style approaches
- Platonic Representation Hypothesis — the theoretical framing that makes shared embedding geometry plausible
Our contribution is engineering — library, CLI, quality gate, adapters, and benchmarks — not algorithmic novelty. If you are citing the underlying technique, cite that work; if you are citing the tool, cite this repo.
The gate lifecycle
Fit the mapping, gate it, migrate, and keep the old store until the verdict says it is safe. The gate is a checkpoint, not a feature bolted on at the end.
1. Calibrate
Embed the same ~2K in-domain texts with the source and target models, fit a ridge map, and save a .isotrieve artifact. See the Quick Start.
2. Gate
Run the quality gate on held-out samples. The verdict is driven by a model trained across benchmark runs, not a hand-tuned threshold. See the Quality Gate docs.
| Verdict | Predicted Retention | Action |
|---|---|---|
| PASS | ≥ 0.75 | Safe to migrate |
| WARN | 0.55 – 0.75 | Proceed with monitoring |
| FAIL | < 0.55 | Do not ship; fix calibration or re-embed |
3. Migrate
Transform the store in place with a migration that tracks progress in a manifest, so you can resume or roll back. The TypeScript migrateStore shows the lifecycle contract; the Python migrate_store is the production path.
import { migrateStore } from '@isotrieve/core';
const manifest = migrateStore(sourceStore, targetStore, mapping, {
batchSize: 1024, // vectors per batch
manifestPath: './manifest.json',
resume: true, // skip already-migrated batches
onBatchComplete: (b) => {
console.log(`Batch ${b.batchNum}: ${b.rowsProcessed} vectors`);
},
});
console.log(manifest.rowsTotal, manifest.rowsMigrated);4. Verify
Keep the old store until spot-checks and the gate pass on the new collection. A manifest makes rollback a replay, not a rebuild. See the migration docs for the full manifest contract.
When NOT to Use Isotrive
Isotrive is a cost/quality tradeoff, not a free lunch. Skip it (or re-embed instead) when:
- Maximum quality matters more than cost — full re-embedding still wins on nDCG when the budget allows
- K < 2000 — with fewer calibration pairs the gate usually lands in WARN; gather more in-domain pairs first
- Calibration domain mismatches the corpus — out-of-domain pairs learn the wrong map, and no gate can fix a map learned on the wrong data
- The gate returns FAIL — that is the answer, not an obstacle. Fix the calibration or re-embed.
Full details, including the When NOT to Use section of the main docs and the anti-patterns, cover the rest of the edge cases.