ShipGauge

Eval suites

Define suites, run scorers, and upload results with the CLI

Eval suites are YAML (or JSON) collections of cases scored for quality and cost. Results upload to your project and feed the CI gate.

Prerequisites

  • Project + ingest API key
  • @shipgauge/cli installed
npm i -D @shipgauge/cli
npx shipgauge init

Config sketch

.shipgauge/config.yml:

api_key: ${SHIPGAUGE_API_KEY}
project_id: ${SHIPGAUGE_PROJECT_ID}
base_url: https://shipgauge.dev
suites_dir: .shipgauge/suites

Suite example

# .shipgauge/suites/smoke.yaml
name: smoke
cases:
  - id: hello
    input: "Say hello"
    expected: "hello"
    scorers:
      - type: contains
        value: hello

Use built-in scorers from @shipgauge/scorers (contains, regex, cost ceilings, etc.) and custom scorers where configured in the project.

Run locally

npx shipgauge eval --suite smoke
# skip upload while iterating:
npx shipgauge eval --suite smoke --no-upload

Uploaded runs appear under Evaluations in the dashboard.

Tips

  • Keep feature_id on cases aligned with production instrumentation.
  • Start with a small smoke suite; grow regressions per feature.
  • CI should run the same suite names as local.

Next