Isotrieve Documentation

Install the core package from PyPI. Adapter extras pull in optional dependencies only when you need them — keep the core install small for CI and production images that only load a saved mapping.

Use isotrieve[chroma], isotrieve[langchain], or isotrieve[qdrant] for a single store, or isotrieve[all] when you want every adapter available in one environment (local demos, notebooks).

• Core — mapping, serve, gate, CLI
• chroma / langchain / qdrant — store-specific wrappers
• all — core + every adapter

Requires Python ≥ 3.10, numpy ≥ 1.24, and scikit-learn ≥ 1.3. After install, confirm the package imports and print the version — this is the quickest sanity check before calibrating.

If import fails, check that you are in the intended virtualenv and that the Python on PATH matches the one used for pip install.

Calibration pairs are the same texts embedded with the source and target models. Aim for at least ~2K in-domain examples for a PASS gate verdict; fewer often works for prototyping but retention drops.

RidgeMapping with alpha="auto" tunes regularization via leave-one-out GCV. After fit, save writes a .isotrieve file (forward map, optional inverse, metadata). transform returns L2-normalized vectors in the target space.

• Works for same-dim and rectangular (d_src ≠ d_tgt) pairs
• Seed the fit for reproducibility across machines
• Reuse the saved file in serve mode or offline migrate

Serve mode leaves the existing index untouched. Embed the query with the new model, then map it into the legacy space with the inverse transform and search the store you already have.

This is the lowest-cost path when re-embedding millions of documents is expensive or when you want to A/B a new model before committing to an offline migration. The mapping file must include a fitted inverse (QueryAdapter.load will error if it does not).

• No corpus rewrites; only query-path latency for the map
• Same .isotrieve artifact as offline transform
• Optional score recalibration if you fitted one

When you are ready to move the corpus into the new space, transform stored vectors in batch and write a new store. Queries then use the new model directly with no per-request map.

Prefer offline migration once serve-mode quality looks good and you want simpler serving (no inverse at query time). Keep the old store until the quality gate and spot-checks pass on the new collection.

• One-time cost proportional to corpus size
• Serve path becomes a normal new-model embed + search
• Use adapters (migrate_collection, etc.) for store-native I/O

The CLI covers the same lifecycle as the Python API without writing a script: estimate cost and shape of a migration, fit from numpy dumps, transform a directory of vectors, and inspect a saved artifact.

plan is useful before you spend on embeddings — pass source/target model names and corpus size to get a rough sense of work. calibrate expects paired .npy matrices. transform walks a source directory and writes mapped vectors. inspect prints header metadata (dims, type, whether an inverse exists).

• plan — migration sketch from model names + corpus size
• calibrate — fit X.npy / Y.npy → .isotrieve
• transform — batch-map a vector directory
• inspect — dump artifact metadata

Ridge regression minimizes ||X W − Y||² + α||W||². With alpha="auto", α is chosen by leave-one-out GCV. An affine bias column is included by default so a pure scale/shift between spaces is absorbed cleanly.

Rectangular pairs (e.g. MiniLM 384d → bge-large 1024d) are first-class: the matrix shape is (d_src (+1), d_tgt). The inverse is fitted separately with its own α so serve mode does not reuse a poorly conditioned transpose.

• Handles rectangular mappings (d_src ≠ d_tgt)
• Bias column for the affine term
• Independent inverse α (fitted separately)
• Default choice on BEIR (higher retention than MLP / low-rank)

Orthogonal Procrustes finds the nearest rotation (orthogonal matrix) aligning two point clouds of the same dimensionality. Use it when you want a volume-preserving map and can guarantee d_src == d_tgt.

It will not handle rectangular migrations. On same-dim BEIR pairs it is competitive but usually slightly behind Ridge; prefer Ridge unless you have a geometric reason to constrain to SO(d) / O(d).

• Requires d_src == d_tgt
• Orthogonal constraint — no shear / scale freedom beyond rotation
• Good for interpretability experiments; Ridge for default quality

QueryAdapter loads a .isotrieve file and exposes query-time mapping into the legacy space. Embed with the new model in your app, call map_query / map_queries, then search the existing index.

Inspect d_src / d_tgt to assert wiring, and has_recalibrator before calling recalibrate_scores. If the inverse was not fitted at calibrate time, load raises — re-fit with inverse enabled rather than shipping a half-complete artifact.

• map_query — single vector; map_queries — batch
• Zero corpus writes; only the query path changes
• Same artifact used for offline transform

Linear maps preserve neighborhood structure better than absolute score scales. Thresholds tuned on the original model (e.g. “accept if cosine > 0.82”) often shift after mapping. Fit an isotonic recalibrator on held-out pairs so serve-time scores land back on a comparable scale.

Call fit_recalibrator after fit, then save so the recalibrator travels with the mapping. At serve time, run retrieval as usual and pass raw scores through recalibrate_scores before applying product thresholds.

MAE and margin compression on SciFact held-outs:

Not every mapped query is equally trustworthy. ConfidenceScorer looks at top-score margins and labels the result high, medium, or low so you can fall back to re-embed, escalate to a stronger ranker, or hide low-confidence hits.

Margins below are tuned from SciFact runs; start there and retune on your domain if the high/low split does not match your risk tolerance. is_reliable is a convenience for “high only” gates.

Before you cut over production traffic, run the quality gate on held-out source/target samples. It predicts retrieval retention and returns PASS, WARN, or FAIL so you do not discover a broken map only after users complain.

Verdicts are driven by a model trained across benchmark runs (including leave-one-pair-out style evaluation). Treat FAIL as a hard stop: gather more or better calibration text, or re-embed. WARN means proceed with monitoring and tighter confidence thresholds.

IsotrieveChromaFunction wraps your new-model embedder so Chroma queries are mapped into the space of the collection you already built. Point an existing collection at this embedding function and keep serving without rewriting rows.

Pass the fitted mapping plus a callable (or object) that embeds text with the new model. Chroma calls that path on query; the adapter applies the inverse map before the distance computation against stored legacy vectors.

• Drop-in for serve mode on an existing collection
• No need to duplicate the corpus under a new name first
• Pair with the quality gate before flipping production

When you want the corpus itself in the new space, migrate_collection reads the source collection in batches, applies transform, and writes a new collection. Tune batch_size for memory; 1000 is a reasonable default for medium-dim vectors.

Keep the old collection until spot-checks and the gate look good on docs_v2. The report’s rows_processed count is useful for ops logging and resume checks.

• Creates new_collection; does not mutate the source in place
• Batch size trades throughput vs peak RAM
• Use after serve-mode validation when possible

IsotrieveEmbeddings implements the LangChain embeddings interface around a base embedder and your fitted map. Document and query paths both go through the adapter so vector stores built with LangChain stay consistent.

Use it when your stack already centers on LangChain retrievers / Chroma / FAISS wrappers. Sync and async embed methods are supported; prefer the same mapping artifact you would use with the plain Python API.

• embed_documents(texts) — corpus / indexing path
• embed_query(text) — single query
• aembed_documents(texts) — async batch

QdrantStore connects to a running Qdrant instance and streams vectors in batches so you can apply m.transform without loading the whole collection into memory.

Typical offline flow: iterate batches, stack to a matrix, transform, upsert into a new collection (or overwrite after validation). Choose batch_size based on vector dim and available RAM — 1024 is a starting point for ~1k-d embeddings.

• Streaming iterator — safe for large collections
• Works with any fitted mapping that implements transform
• Pair with gate + spot queries before deleting the old collection

A .isotrieve file is a JSON header followed by float64 matrix payloads. The header records format version, mapping type, dimensions, whether a bias column was used, whether an inverse exists, and the forward matrix shape.

Payloads carry the forward W and, when present, W_inv. Optional recalibrator state is stored alongside so serve mode can restore score calibration without a second file. Treat the artifact as opaque for day-to-day use — prefer Mapping.load / inspect over hand-editing.

• Header is human-readable; matrices are binary float64
• Forward and inverse travel together when fitted
• Bump-aware via format_version