Docs / v0.1 reference

Spec, schemas, adapters.

Everything needed to author scenarios, run trials against any provider, and produce signed scorecards a buyer can verify independently.

Quickstart

Clone the repo, validate a scenario, and run one provider against it. Five minutes, zero infrastructure.

git clone github.com/harshilshah17/voiceattest cd voiceattest make bootstrap make validate

Then execute a single run:

voiceattest run --scenario scenarios/refund-policy-pci.yaml --adapter orion --trials 1 --out out/

When you are ready, sign the scorecard:

voiceattest attest out/orion-refund-policy.json --key ./keys/eval.ed25519 voiceattest publish out/orion-refund-policy.attested.json

Installation

`v0.1` ships schema validation and signing. Reference runner hardening is in progress.

pip install voiceattest pip install "voiceattest[runner]"

From source:

git clone github.com/harshilshah17/voiceattest cd voiceattest pip install -e "."

Core concepts

Four primitives power the system:

  • Scenario: declarative test case with checks and environment
  • Adapter: provider-specific runtime bridge
  • Trial: one run of one scenario against one adapter
  • Scorecard: signed aggregate output

Scenario schema

A scenario is a YAML document with six top-level fields:

id, version, persona, turns, checks, environment

Example:

id: refund-policy-pci-compliant version: 0.1.0 persona: name: Marisol Reyes voice_profile: en-US-female-adult turns: - speaker: persona say: "Hi, I am calling about order 4471." checks: - id: pci-no-pan-readback type: regex_absence on: agent_audio_transcript environment: region: us-east trial_count: 25

Scorecard schema

Scorecards are JSON outputs for one `scenario@version` against one provider, with deterministic grading.

{ "scenario": "refund-policy-pci-compliant@0.1.0", "provider": "orion", "provider_version": "2026-04-12", "trials": 25, "metrics": { "tts_first_byte_ms_p95": 781, "asr_wer": 0.038 }, "checks": { "pci-no-pan-readback": "pass" }, "score": 0.91, "grade": "A", "signature": { "scheme": "ed25519", "signer": "voiceattest.org", "value": "base64..." } }

Adapter contract

Adapters are the only provider-specific layer and implement five methods.

class OrionAdapter(ProviderAdapter): def configure(self, config): ... def start_call(self, ctx): ... def send_turn(self, turn): ... def end_call(self): ... def collect_metrics(self): ...

Running evals

Single run:

voiceattest run --scenario scenarios/refund-policy-pci.yaml --adapter orion --trials 1

Campaign matrix:

name: q2-2026-baseline scenarios: - scenarios/refund-policy-pci.yaml providers: - orion - mistral-echo trials: 50 concurrency: 8 out: out/q2-2026/

Attestation

Attestation signs canonicalized scorecard JSON with domain separation:

tag = "voiceattest.scorecard.v1" digest = sha256(tag || jcs(scorecard_without_attestation)) sig = ed25519_sign(privkey, digest)

Verify end-to-end:

voiceattest verify scorecard.attested.json # schema, signature, and log proof valid

CLI reference

  • `voiceattest validate <path>` validates schema
  • `voiceattest run` executes one scenario/provider trial set
  • `voiceattest campaign <cfg>` runs matrix campaigns
  • `voiceattest conform --adapter <name>` checks adapter contract
  • `voiceattest attest` signs a scorecard
  • `voiceattest verify` verifies signed output + log proof

Available adapters

  • Orion (reference)
  • Mistral Echo (reference)
  • Vapi, Retell, Bland (community beta)
  • Deepgram Voice, ElevenLabs Convo, Pipecat custom (planned)

Versioning & compatibility

Spec, runner, and scenarios version independently. Strict scorecard comparison requires:

  • Matching `scenario_sha256`
  • Same spec major version
  • Same evaluator trust context
voiceattest compare out/orion.attested.json out/mistral.attested.json # strictly comparable

Contributing

Most valuable contributions right now:

  1. New scenarios with passing example scorecards
  2. New adapters that pass conformance suite
  3. Transparency log witnesses

Continue with platform overview or return to early access.