# Budgets & Cost Management (goose lab)

How **Solo Enterprise AgentGateway** budgets are wired on k8s-goose, how to run the
multi-team demo, and how to read **Within / Exceeding** in the Solo UI.

**Live (styled):** [LLM gateway → Budgets](https://goose.maniak.ai/llm-gateway.html#budgets)

## Overview

| Path | Auth | Budget resources | UI / demo goal |
|------|------|------------------|----------------|
| `/budget-demo` | none | `budget-demo/budget-demo` | Hard **Block** 429 (~2k tokens/day) |
| `/vk-openai` | Strict virtual key | `team-budget-platform`, `team-budget-sales`, `team-budget-marketing` (+ `cost-hierarchy`) | Three named team budgets · Within vs Exceeding |
| `/openai` | none | — | kagent default; priced, not budget-enforced |

Windows are **rolling Day** (rate-limit infra), not calendar midnight.

## Multi-provider budgets (OpenAI + xAI + DGX)

**Yes — one budget can mix providers.** AgentGateway budgets are not “OpenAI-only.” A request
counts when it hits a route with `entBudgetEnforcement` and matches the budget `subject`
(or the entry has no subject). Built-in dimensions include **`provider`** and **`model`**.

| `subject` example | Effect |
|-------------------|--------|
| `group: platform` only | Team spend across **every** enforced provider path |
| `provider: openai` | OpenAI only |
| `provider: xai` / model match | Grok (or that model) only |
| *(empty subject)* | All traffic on routes that attach the Budget |

**Pricing vs budgeting**

| Provider | Catalog rates (per 1M tokens) | Team budget enforced on goose? |
|----------|-------------------------------|--------------------------------|
| OpenAI (`/vk-openai`) | catalog (gpt-5.5, …) | **Yes** — `team-budget-*` |
| OpenAI (`/openai`) | yes | No |
| xAI Grok (`/grok`) | **grok-4.5 $2.00 / $6.00** (cache $0.50); grok-4.3 $1.25 / $2.50 | **No** (not on VK + enforcement yet) |
| Google Gemini (`/gemini`) | gemini-2.5-flash $0.30 / $2.50; pro $1.25 / $10.00 | **No** |
| Anthropic (`/anthropic`) | claude-fable-5 $10 / $50 | **No** |
| DGX Qwen (`/spark`) | **$0.15 in / $0.25 out** | **No** (not on VK + enforcement yet) |

To make `team-budget-platform` a true OpenAI+xAI+DGX mix, attach the same virtual-key auth +
`entBudgetEnforcement` (Same ns) to those routes (or add `/vk-grok` / `/vk-spark`). Until then,
team budgets only accumulate **OpenAI `/vk-openai`** spend; other providers still appear on
**Dashboard** spend-by-model charts.

## Discovery isolation (important)

`entBudgetEnforcement` loads budgets from the namespaces it is configured to watch.
**Subject-less** budget entries match **every** request on any route that has enforcement.

If `demo-token-block` (2000 tokens, **Block**) lives in `agentgateway-system` next to
`/vk-openai` enforcement, multi-team VK traffic will eventually get **HTTP 429** even when
team budgets are only **Audit**.

**Lab fix:**

- Budget CR `budget-demo` → namespace **`budget-demo`** (label `goose.maniak.ai/budget-ns=budget-demo`)
- Policy `budget-demo-enforcement` (on HTTPRoute `budget-demo`) uses:

  ```yaml
  entBudgetEnforcement:
    discovery:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            goose.maniak.ai/budget-ns: budget-demo
  ```

- Policy `openai-vk-cost` uses `from: Same` → only `agentgateway-system` budgets
  (`team-budget-platform|sales|marketing`, `cost-hierarchy`).

Manifest: `config/policies/budget-demo.yaml`.

## GitOps files

| Path | Role |
|------|------|
| `config/policies/budget-demo.yaml` | NS + Block demo budgets + selector enforcement |
| `config/routes/budget-demo-route.yaml` | HTTPRoute `/budget-demo` |
| `config/policies/team-budget-demo.yaml` | **3 CRs:** `team-budget-platform`, `team-budget-sales`, `team-budget-marketing` |
| `config/policies/cost-hierarchy-budgets.yaml` | Example org/group/user budgets |
| `config/policies/openai-vk-cost.yaml` | VK auth + budget enforcement on `/vk-openai` |
| `config/routes/openai-vk-route.yaml` | HTTPRoute `/vk-openai` |
| `config/secrets/goose-virtual-keys.yaml` | 9 keys (3 teams × 3 users) in both namespaces |
| `config/ui-cost-shim/agentgateway-enterprise-budget-dimensions.yaml` | UI hierarchy dimensions |
| `config/policies/model-cost-catalog.yaml` | USD rates for pricing |
| `scripts/test-budget-teams.sh` | Multi-team traffic burn |
| `scripts/test-cost-vk.sh` | Lighter VK smoke |

Applied by ArgoCD app **`agentgateway-config`** (path `config/`).

## Virtual keys (users · teams · **client / IDE**)

Secret label: `agentgateway.solo.io/virtual-key-set: goose-virtual-keys`.

Hierarchy in Cost Management:

`organization → costCenter → group → user → **client**`

| Budget CR | Team (`group`) | Cost center | Users | Bearer |
|-----------|----------------|-------------|-------|--------|
| **team-budget-platform** | platform | engineering | alice, bob, chris | `sk-goose-platform-<user>` |
| **team-budget-sales** | sales | gtm | dave, erin, frank | `sk-goose-sales-<user>` |
| **team-budget-marketing** | marketing | growth | grace, heidi, ivan | `sk-goose-marketing-<user>` |

### Client / IDE keys (same user, different `client` dimension)

| Key name | Bearer | user | **client** | Use for |
|----------|--------|------|------------|---------|
| alice-claude-code | `sk-goose-alice-claude-code` | alice | **claude-code** | Claude Code IDE |
| alice-opencode | `sk-goose-alice-opencode` | alice | **opencode** | OpenCode IDE |
| platform-alice | `sk-goose-platform-alice` | alice | **lab-api** | scripts / harness |

Organization: `solo-io`. Metadata: `id`, `organization`, `costCenter`, `group`, `user`, **`client`**.

```bash
# Lab script traffic (client=lab-api)
curl -sS http://172.16.10.155:30160/vk-openai/v1/chat/completions \
  -H "Authorization: Bearer sk-goose-platform-alice" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4.1-nano","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}'

# Generate Claude Code vs OpenCode spend in Cost UI
./scripts/test-cost-vk.sh clients
```

**IDE config (OpenAI-compatible):** base URL `http://172.16.10.155:30160/vk-openai/v1` · API key = the bearer above.

## Budget CRs (one per team in the UI list)

Each file entry is a **separate** `EnterpriseAgentgatewayBudget` so Solo UI shows three rows.

| CR name | Team limit | Extra user limit | Intent after full burn |
|---------|------------|------------------|------------------------|
| `team-budget-platform` | group=platform · **500** tokens/day | alice · 200 | **Exceeding** |
| `team-budget-sales` | group=sales · **450** tokens/day | dave · 180 | **Exceeding** |
| `team-budget-marketing` | group=marketing · **400** tokens/day | grace · 160 | **Exceeding** |

All **Audit** + rolling **Day** (spend can pass the limit so the UI shows red without 429).

## Run multi-team UI demo

```bash
cd /path/to/k8s-goose

./scripts/reset-budget-demo.sh          # clear live counters
./scripts/test-budget-teams.sh status
./scripts/test-budget-teams.sh          # full burn
```

Then:

1. Open Solo UI: `https://172.16.10.155:30443` (Entra login)
2. Product **AgentGateway** → **Cost Management** → **Budgets**
3. You should see **`team-budget-platform`**, **`team-budget-sales`**, **`team-budget-marketing`**
4. Chips: **Exceeding budget** count climbs; open each CR for entry bars
5. Optional: Scope by hierarchy (platform / sales / marketing groups)

Allow ~15–30s for usage to refresh.

## Hard Block path (`/budget-demo`)

```bash
for i in 1 2 3 4 5 6 7 8; do
  curl -sS -o /tmp/bd.json -w "try $i → HTTP %{http_code}\n" \
    -X POST http://172.16.10.155:30160/budget-demo/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"gpt-5.5","max_tokens":200,"messages":[{"role":"user","content":"Write two paragraphs about geese."}]}'
done
```

Expect eventual **HTTP 429** when `demo-token-block` is exceeded. Does **not** require a virtual key.

## Clear / reset live budget counters

```bash
./scripts/reset-budget-demo.sh              # flush Redis + restart RL/proxy
./scripts/reset-budget-demo.sh --dry-run
./scripts/reset-budget-demo.sh --wipe-clickhouse-demo   # also truncate cost rollups
```

Then re-run `./scripts/test-budget-teams.sh`. Does **not** delete Budget CRs or virtual keys.

## Troubleshooting

| Symptom | Check |
|---------|--------|
| Every `/vk-openai` call is 429 | Run `./scripts/reset-budget-demo.sh`. Stale subject-less Block still in `agentgateway-system`? Budgets for Block demo should only be in ns `budget-demo`. |
| UI budgets empty / zero usage | Hit `/vk-openai` with a real key + enforcement attached. Time range 24h. |
| 401 on VK path | Secret `goose-virtual-keys` (12 keys); restart proxy after Secret replace. |
| Never “Exceeding” | Run full `test-budget-teams.sh`; limits are token Day + Audit; need enough in+out tokens. |
| Hierarchy empty | Dimensions CMs + traffic with custom dimensions (VK + enforcement). |

## Related

- [LLM gateway lab](https://goose.maniak.ai/llm-gateway.html) (features, configs dropdown, scenarios)
- [MCP gateway lab](https://goose.maniak.ai/mcp-gateway.html) (tool RBAC — separate plane)
- [Test harness](./test-harness.md)