Docs

Use IA4AI

Capture, golden sets and verification

How a cheaper model is proven on your own traffic before anyone switches, and what that needs from you.

A cheaper model on a price list is a guess. IA4AI states a saving only once a cheaper model has been run on your own real inputs and shown to answer the way your current model did. That takes three things, in order.

#1. Capture: keep real inputs, for a while

By default IA4AI keeps no prompts or answers, so there is nothing to replay. Capture is how that changes, and it is built to be hard to turn on by accident:

  • Your workspace's admin turns it on, under Content mode: for 1 to 90 days, for the kinds of content chosen (prompts and answers, reasoning, tool arguments), optionally for some agents only, with a stated purpose. It closes by itself, and can be turned off at any moment.
  • Only your own key counts. Content is kept only for calls made with your workspace's key while content mode is on. Anything else is dropped on arrival, whatever it sends.
  • Redacted, then encrypted. API keys, tokens, private keys, email addresses, phone numbers, card numbers and IP addresses are replaced before storage, and the content is encrypted with a key unique to your workspace. The redaction is pattern-based: names and free-form personal details are not recognised.
  • Bounded and temporary. A prompt or answer over 256 KiB keeps only its hash. Captured content is deleted after 30 days by default, even when the usage records are kept longer.

More on content mode, export and erasure: Privacy and your data.

#2. Golden sets: fix the inputs

A golden set is a fixed list of your captured inputs with the answers your current model gave. A verdict means something only next to the exact cases it was measured on, so each set is content-addressed: its version is a hash of its cases, and every verdict names the version it was measured on.

Terminal
curl -s -X POST https://ai.odux.io/app/golden-sets -H "x-api-key: YOUR_IA_KEY" \
  -H 'content-type: application/json' -d '{"description": "baseline before routing changes"}'
  • A set holds only your own workspace's traffic, at most 100 cases by default, optionally one job's.
  • Listing sets never returns content; reading a set's cases is audited.
  • A set's content expires with the oldest capture in it. The record (version, counts, dates) stays, so an old verdict still says what it was measured on.
  • A 422 with stats means there was no captured content to build from, and says why.

#3. Verification: replay and compare

A verification run replays a golden set on your current model and on each cheaper candidate, and a judge compares every answer with the one your current model gave in production. The verdict for each candidate is one of:

Verdict Means
Verified A swap you can make: the cheaper model answered the same way, across several seeds.
Unproven A tie. More cases are needed before anything is claimed.
No The cheaper model answers differently.
Terminal
curl -s -X POST https://ai.odux.io/app/case/<case_id>/verify -H "x-api-key: YOUR_IA_KEY" \
  -H 'content-type: application/json' -d '{"set_id": "<set_id>"}'
curl -s https://ai.odux.io/app/verification-runs/<run_id> -H "x-api-key: YOUR_IA_KEY"

What keeps a verdict honest:

  • Your keys pay, within a budget you agree. Runs use your own stored provider keys, only for models those keys may call. The daily budget is $0 (off) until we set it for your workspace at your request.
  • The worst case is checked first. A run whose worst-case cost exceeds what is left of today's budget is refused (409) before anything runs. A run that would go over stops, with no verdict.
  • The current model runs again. Your current model answers the same inputs again, so the comparison is fair even though captured inputs are redacted.
  • A conservative judge. By default the judge is your current model. A model judging its own answer leans towards it, so any bias works against declaring a cheaper model equal.
  • Tool calls are left out, and counted. Turns that called tools are not replayed, because their tool definitions were not captured; the report says how many were left out. A run uses at most 20 cases.
  • On the record. Starting a run is audited, and its outcome (status, cost, case, golden-set version) goes into the evidence log.

Verified swaps show on the case page and in your plan, under Verified on your own traffic. A verdict older than 30 days is shown as stale. The headline totals stay labelled as estimates: a few verified cases do not make a fleet-wide estimate a fact.

#Live side-by-side tests

Separately from verification, the gateway can send a small sample of live calls to a cheaper model as well, and compare the two answers. The cheaper-model tests report shows, per pair, how many calls were sampled and matched, and says plainly when a key's model list blocks the test. It states no saving either: it is the evidence that tells you which candidate is worth verifying.