Concepts
Features, gates, budgets, and what ShipGauge stores
Organization vs project
- Organization — billing, team seats, plan limits, SSO.
- Project — instrumentation surface: API keys (project-scoped), events, eval suites, budget rules, integrations.
Most SDK calls are scoped to a project id.
feature_id
Stable string naming the product capability you want to measure and gate, e.g. chat, summarize, support-reply.
Use the same id in SDK tags, eval cases, and budget scopes so Overview, Margin, and CI stay aligned.
Environments
Events carry a free-text environment label. Common values:
| You send | Dashboard treats as |
|---|---|
production, prod, live | Production |
staging, stage, stg | Staging |
preview, preprod | Preview |
development, dev, local | Development |
| anything else | its own filter chip (e.g. canary) |
If omitted, ingest and the Node SDK default to production. Prefer passing your host env (e.g. process.env.VERCEL_ENV) so preview deploys do not mix with production spend.
The Overview / Features / Margin / Customers views honor the env switcher (aliases match the same filter).
Ingest API keys
Keys look like sg_ingest_.... They authenticate:
POST /api/v1/events- Eval run upload / gate-check endpoints used by the CLI
Treat them like production secrets. Revoke from Account → API keys if leaked.
Gate vs budget vs alert
| Mechanism | When | Effect |
|---|---|---|
| Eval + gate | CI / pre-merge | Fail the job or required check when quality/cost regress vs baseline |
| Soft budget | Production | Notify (email/Slack) as you approach a limit |
| Hard budget | Production | Block further spend (SDK/HTTP signal) when the limit is hit |
Gates answer “should we ship?”. Budgets answer “can this feature keep spending?”.
Scorers (high level)
Eval suites score cases (pass/fail, cost, latency, custom scorers). Gate rules aggregate suite results: minimum pass rate, max cost increase, etc. Details: Eval suites and CI Gate.
What we store
| Stored | Not stored |
|---|---|
| Model, provider, tokens, estimated cost, latency | Prompt / completion text |
feature_id, optional customer_id, hashes | Raw PII you never send |
| Eval scores and gate reports | Full production traces (unless you export OTel elsewhere) |