How Isotrieve Works - Embedding Migration Protocol

1. Calibration

Embed ~2K texts with both the source and target models. This creates paired matrices X (source) and Y (target).

# Embed K calibration texts with both models
X = source_model.embed(calibration_texts)  # (K, d_src)
Y = target_model.embed(calibration_texts)  # (K, d_tgt)

Calibration texts should come from your corpus domain. Out-of-domain degrades quality.

2. Fit Mapping

Ridge regression finds the optimal linear transformation:

W = argmin ||X · W - Y||² + α||W||²

α is auto-tuned via leave-one-out GCV. Handles both same-dimension and rectangular pairs.

from isotrieve import RidgeMapping

m = RidgeMapping(alpha="auto", seed=0)
m.fit(X, Y)
m.save("mapping.isotrieve")

The .isotrieve file stores: forward W, inverse W_inv (independent α), optional recalibrator, metadata.

3. Serve (query-time mapping)

No corpus writes needed. Map new-model queries into legacy space on-the-fly:

New Modeld_tgt dim

→W^-1

Legacy Spaced_src dim

from isotrieve.serve import QueryAdapter

qa = QueryAdapter.load("mapping.isotrieve")
legacy_vec = qa.map_query(new_model_embed(query))
results = legacy_index.search(legacy_vec, k=10)

4. Offline Migration (optional)

Transform stored corpus vectors. Source collection is never modified.

Legacy Vectorsd_src dim

→W

New Model Spaced_tgt dim

from isotrieve.adapters.chroma import migrate_collection

report = migrate_collection(client, "docs", mapping)
print(f"Migrated {report.rows_processed} rows")

Math

Forward: Y ≈ X · W (ridge regression with bias column)
Inverse: X ≈ Y · W_inv (separate α, independently fitted)
Normalization: All outputs are L2-normalized
Complexity: O(K · d²) fit, O(N · d · d') transform

What's Been Tested

Operation	Result
Ridge (default)	86.6% retention (MiniLM→bge, K=4000)
Procrustes	85.1% retention (same-dim, square only)
Independent inverse α	+2.2pts improvement
Score recalibration	Essential for rectangular pairs (+22% at τ=0.60)
Cross-encoder reranking	NULL RESULT (−10.7pts)
TSVD shrinkage	NULL RESULT (−0.33pt at rank=512)
Procrustes centering	REVERTED (−55pt on unit vectors)

Resource Requirements

Operation	Time	Memory
Fitting (K=4000)	~0.5s	~50 MB
Transform (1M vectors)	~10s	Streaming
Query mapping	<1ms	<1 MB

Isotrieve vs. Re-embedding

Metric	Re-embedding	Isotrieve (mapped)	Isotrieve (serve)
Retention	100%	86–92%	86–92%
Corpus writes	Yes (all docs)	Yes (once)	No
Rollback	Slow	Delete collection	Instant
Cost	O(N)	O(K)	O(K)