ShipGauge

CI Gate

Fail merges when pass rate, cost, or latency regress

shipgauge gate check compares the latest eval suite against a baseline and exits non-zero on regression.

When to use

  • Pull requests that change prompts, models, or retrieval
  • Release candidates where cost creep matters as much as quality

Gate rules

.shipgauge/gate.yml example:

rules:
  pass_rate:
    min: 0.95
    max_drop: 0.05
  avg_cost_usd:
    max_increase: 0.20
  fail_on_any_case: true

bypass_label: shipgauge-skip

Tune thresholds to your product risk — tighter for money paths, looser for experimental agents.

Run

npx shipgauge gate check --suite smoke

Useful flags (CLI):

  • --status-check — create/update GitHub check run ShipGauge Gate
  • --pr-comment — upsert a PR summary comment
  • --no-upload — local-only comparison when developing rules

Baseline

The first CI run without a baseline typically records the current suite as baseline and passes. Subsequent runs compare against that snapshot. Re-baseline deliberately when you intentionally ship a new quality/cost profile.

Verify

  • Local: exit code 0 / non-zero
  • GitHub: check named exactly ShipGauge Gate
  • Dashboard: recent eval run + gate history

Next