I built Sentry for AI costs
Last year I got an OpenAI bill for $2,100. I knew my app made AI calls. I didn't know which feature was responsible for 80% of that bill.
The billing dashboard told me I used GPT-4. Great. But was it the search feature? The summarizer? The nightly batch job that someone forgot to turn off? No way to tell.
I looked at existing approaches. Some require wrappers, manual tags, or a gateway in the request path. I wanted to keep provider clients and routes in the stack I already had.
I didn't want that. I wanted what Sentry did for errors: add one line, see everything.
How CostKey works
CostKey patches fetch (Node.js/TypeScript) and httpx (Python) at the runtime level. When your app makes a call to a supported provider endpoint, CostKey observes the response, extracts usage and timing when the response shape is covered, captures stack context, and sends telemetry to the dashboard.
The key insight: the stack trace tells you which function made the call. CostKey walks the stack frames, filters out library code (node_modules, site-packages), and identifies the "business logic" function that actually triggered the AI call.
// This is all you add to your app
import { CostKey } from 'costkey'
CostKey.init({ dsn: 'http://ck_...@localhost:4100/proj_...' })
// Everything below is your existing code, unchanged
const result = await openai.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: query }]
})
Initialize once. Existing provider clients and routes remain in place. The SDK is MIT licensed, and talk to us about pricing, or run it yourself: the platform is MIT licensed too.
What you see
The dashboard shows you:
- Cost per function: connect a captured charge to the function that triggered it
- Cost per deploy: compare release-tagged work when release context is supplied
- Request traces: inspect the calls, models and costs inside one operation
- Spend findings: investigate a change with its supporting activity
- Streaming metrics: inspect timing when the endpoint exposes the needed stream data
The attribution algorithm
The hardest part was getting the function attribution right. When your code calls openai.chat.completions.create(), the stack trace has many frames, most of them library internals. We need to find your function, not OpenAI's.
The algorithm: filter frames from known library paths (node_modules/openai, site-packages/anthropic, etc.), skip known wrapper function names (retry, withBackoff, etc.), then find the first frame from a different file than the API call. That's your business logic function.
In Python, we go further: the SDK scans your codebase at init using AST analysis to build a call graph. Functions with high "fan-in" (called by many other functions) are likely wrappers. Functions with low fan-in are likely business logic. This empirical ranking improves attribution accuracy.
Attribution remains a runtime capability, so source maps and application context can improve what the dashboard can explain.
Why not a proxy?
A gateway can be the right fit when routing or centralized policy is the job. CostKey takes a different path: it observes supported calls in the application so existing provider clients and routes can stay in place.
CostKey observes the response after the provider call returns and queues telemetry in the background. Capture, serialization and flushing still add application work, so profile your runtime. CostKey's own delivery failures are kept separate from provider errors, which remain visible to your application.
Try it
Talk to us about pricing, or self-host it. Provider consumption and the CostKey subscription remain separate. Capacity and operating economics are measured before any unbounded service promise.
# Python
pip install costkey
import costkey
costkey.init(dsn="your-dsn-here")
# TypeScript
npm install costkey
import { CostKey } from 'costkey'
CostKey.init({ dsn: 'your-dsn-here' })