Tracking LLM Cost Per Feature: A Developer's Guide

As AI applications scale from prototypes to production, engineering leaders and indie founders face a common operational challenge: "Our monthly LLM invoice is $3,500, but which features are driving that cost, and is each feature actually generating positive ROI?"

Lumping all OpenAI or Anthropic API calls into a single global billing bucket creates a major blind spot. Without feature-level attribution:

• High-value core features (e.g., automated code generation) may be starved of budget.
• Unoptimized side features (e.g., cosmetic auto-tagging) might quietly consume 60% of your token quota.
• Prompt modifications introduced in pull requests can double operational costs without detection until the end of the billing cycle.

In this guide, we'll cover how to tag and attribute LLM calls by feature, calculate unit economics, and automate cost-regression testing in your GitHub Actions CI/CD pipeline.

---

1. Defining Feature Metadata in LLM Invocations

The first step in feature attribution is attaching a clear, consistent feature_id (and optionally environment or user_tier) to every LLM request.

Option A: Custom Headers (Native SDKs)

If you call providers directly, pass tracking identifiers via request headers or metadata options:

// Example using OpenAI Node SDK with custom header tagging
const completion = await openai.chat.completions.create(
  {
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Generate headline..." }],
  },
  {
    headers: {
      "X-Feature-ID": "marketing-headline-generator",
      "X-Environment": "production",
    },
  }
);

Option B: Unified Wrapper SDK (ShipGauge)

Using @shipgauge/node, feature attribution is handled cleanly at wrapper initialization or invocation time:

import { ShipGauge } from "@shipgauge/node";
import OpenAI from "openai";

const sg = new ShipGauge({ apiKey: process.env.SHIPGAUGE_API_KEY });
const openai = sg.wrapOpenAI(new OpenAI());

// Feature ID passed explicitly per call
const completion = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Extract keywords..." }],
  // @ts-ignore - ShipGauge custom feature tracking extension
  shipgauge: {
    featureId: "smart-tagger",
    metadata: { userPlan: "pro" },
  },
});

---

2. Calculating Unit Economics & $/Req Metrics

Once calls are tagged by feature_id, aggregate metrics into three core Key Performance Indicators (KPIs):

1. Total Daily Spend per Feature ($/day): Uncovers which features consume the bulk of the monthly budget.
2. Average Cost per Request ($/req): Identifies inefficiency or bloated prompt context windows.
3. P95 Latency per Feature (ms): Ensures high token counts aren't degrading end-user experience.

Example Feature Cost Breakdown Table:

Feature ID	Daily Volume	Avg Tokens / Req	Avg Cost / Req	Monthly Spend	Status
code-explainer	14,200	2,450	$0.0036	$1,533.60	🟢 High Value
auto-summarizer	85,000	480	$0.0002	$510.00	🟢 Optimized
smart-tagger	120,000	3,100	$0.0046	$16,560.00	🔴 Needs Attention

Notice how smart-tagger costs 10x more than expected due to an uncompressed system prompt included on every single request.

---

3. Automating Cost-Regression Guardrails in CI/CD

Tracking costs in production is essential, but preventing cost regressions before code hits production is even better.

When an engineer modifies a prompt or changes a model parameter in a Pull Request, your CI pipeline should automatically run eval suites and report the cost difference right in the PR comments.

GitHub Action Workflow (.github/workflows/gate.yml)

name: ShipGauge Cost & Quality Gate

on:
  pull_request:
    branches: [main]

jobs:
  eval-and-gate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Run ShipGauge Eval Suite & Gate Check
        uses: shipgauge/action@v1
        with:
          api-key: ${{ secrets.SHIPGAUGE_API_KEY }}
          gate-config: "./shipgauge.gate.yml"
          github-token: ${{ secrets.GITHUB_TOKEN }}

Automated GitHub PR Comment Output:

When the Action runs, ShipGauge posts an automated PR comparison summary:

### 🛡️ ShipGauge Pre-Ship Gate Results

| Metric | Main Branch | Current PR | Delta | Status |
| :--- | :---: | :---: | :---: | :---: |
| **Pass Rate** | 96.0% | 98.0% | +2.0% | ✅ PASS |
| **Avg Cost / Req** | $0.0012 | $0.0028 | +133.3% | ❌ BLOCK |
| **P95 Latency** | 420ms | 890ms | +111.9% | ⚠️ WARN |

> ❌ **Gate Rejected**: Cost per request increased by >15%. Please optimize prompt length before merging.

---

Conclusion

Feature-level cost attribution shifts your team from reactive billing panic to proactive unit-economic optimization. By tagging every request and enforcing cost gates in CI/CD, you can ship AI features confidently without fear of accidental cost spikes.

Ready to gain full visibility into your LLM costs per feature? Get started with ShipGauge for free today.