One control plane for every model call.

#What AgentGateway does (LLM)

Features exercised in this lab — not a product brochure, the actual CRDs and scripts on goose.

#Deployed config & policy

Everything below is in the k8s-goose repo and applied by ArgoCD (agentgateway-config + control-plane Helm). Pick an item to see kind, path, and open the file on GitHub.

#Architecture

Clients and kagent hit NodePorts / in-cluster services. Gateways price and forward; control plane programs XDS; telemetry lands in ClickHouse for Cost Management.

request path · pricing · cost attribution

#Budgets & Cost Management

How goose wires EnterpriseAgentgatewayBudget, virtual keys, multi-provider spend, and the Solo UI Budgets tab (Within vs Exceeding). Manifests live under config/ · full markdown: docs/budgets.md.

Can one budget mix OpenAI + xAI + DGX?

Yes. Budgets are not bound to a single LLM provider. A request counts toward a budget when it (1) hits a route with entBudgetEnforcement and (2) matches the budget subject (or the entry has no subject). provider and model are built-in dimensions — you can scope a budget to one provider, or leave provider out so team spend mixes every enforced backend.

Paths & named team budgets

discovery isolation · three team CRs on /vk-openai

GitOps manifests

Three named team budgets (UI list)

Each is a separate EnterpriseAgentgatewayBudget so Solo UI shows three rows. Subjects use virtual-key metadata after Strict auth (apiKey.group / apiKey.user).

All Audit + rolling Day (UI can show Exceeding without 429). Org solo-io on every key.

Authorization: Bearer sk-goose-sales-dave
Authorization: Bearer sk-goose-marketing-grace
Authorization: Bearer sk-goose-platform-alice

Run the multi-team demo

1. Clear live counters (optional but clean)./scripts/reset-budget-demo.sh
2. Burn traffic./scripts/test-budget-teams.sh · light / status
3. Open Solo UIHTTPS https://172.16.10.155:30443 (Entra) → AgentGateway → Cost Management → Budgets
4. Open each team CRteam-budget-platform · team-budget-sales · team-budget-marketing — chips Within / Exceeding
5. Optional hierarchyDashboard Scope by hierarchy: org → cost center → group → user · spend by model still shows OpenAI / xAI / Qwen from all routes

./scripts/reset-budget-demo.sh          # flush Redis budget counters
./scripts/test-budget-teams.sh status
./scripts/test-budget-teams.sh          # full burn → all 3 teams Exceeding

# single team request (counts toward team-budget-sales)
curl -sS http://172.16.10.155:30160/vk-openai/v1/chat/completions \
  -H "Authorization: Bearer sk-goose-sales-dave" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","max_tokens":120,"messages":[{"role":"user","content":"budget demo"}]}'

Hard Block demo (/budget-demo)

Separate from teams. No virtual key. After ~2000 tokens in the rolling day window, gateway returns HTTP 429 (onBudgetExceeded: Block).

for i in 1 2 3 4 5 6 7 8; do
  curl -sS -o /tmp/bd.json -w "try $i → HTTP %{http_code}\n" \
    -X POST http://172.16.10.155:30160/budget-demo/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"gpt-5.5","max_tokens":200,"messages":[{"role":"user","content":"Write two paragraphs about geese."}]}'
done

Troubleshooting

#Test scenarios

From the k8s-goose repo root, on a host that can reach 172.16.10.155. No client API keys. Full budget design is in Budgets.

./scripts/test-llm-gateways.sh all

# single provider
./scripts/test-llm-gateways.sh openai
./scripts/test-llm-gateways.sh xai
./scripts/test-llm-gateways.sh claude
./scripts/test-llm-gateways.sh gemini
./scripts/test-llm-gateways.sh dgx
./scripts/test-llm-gateways.sh gpt56

✓ openai   gpt-5.5          HTTP 200  · pong
✓ gpt56    gpt-5.6          HTTP 200
✓ xai      grok-4.5         HTTP 200
✓ claude   claude-fable-5   HTTP 200
✓ gemini   gemini-2.5-flash HTTP 200
✓ dgx      Qwen3.6-35B      HTTP 200  (local)

./scripts/test-llm-lb-failover.sh all
./scripts/test-llm-lb-failover.sh failover-force   # optional deeper demo

# repeat until 429 (demo-token-block = 2000 tokens / day rolling)
for i in 1 2 3 4 5 6 7 8; do
  curl -sS -o /tmp/bd.json -w "try $i → HTTP %{http_code}\n" \
    -X POST http://172.16.10.155:30160/budget-demo/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"gpt-5.5","max_tokens":200,"messages":[{"role":"user","content":"Write two paragraphs about geese."}]}'
done
cat /tmp/bd.json | head -c 200; echo

try 1 → HTTP 200
try 2 → HTTP 200
…
try N → HTTP 429
# body mentions budget / rate limit; proxy log budget_action=BLOCK

./scripts/test-budget-teams.sh status
./scripts/test-budget-teams.sh          # full burn
# UI: https://172.16.10.155:30443 → AgentGateway → Cost Management → Budgets

team-budget-platform    → Exceeding
team-budget-sales       → Exceeding
team-budget-marketing   → Exceeding
# OpenAI /vk-openai only today; xAI + DGX priced but not team-enforced yet
# see #budgets multi-provider section

./scripts/test-cost-vk.sh all

# one persona (3 calls)
./scripts/test-cost-vk.sh platform-alice

# manual
curl -sS http://172.16.10.155:30160/vk-openai/v1/chat/completions \
  -H "Authorization: Bearer sk-goose-platform-alice" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.5","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'

CostUsd 0.00057
CustomDimensions {
  organization: solo-io,
  costCenter:   engineering,
  group:        platform,
  user:         alice,
  virtualKey:   vk-platform-alice,
  provider:     openai,
  model:        gpt-5.5
}

curl -sS http://172.16.10.155:30160/openai/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gpt-5.5","max_tokens":16,"messages":[{"role":"user","content":"ping"}]}'

kubectl --context maniak-goose -n agentgateway-system logs deploy/agentgateway-proxy --tail=5 \
  | grep -o 'agw.ai.usage.cost.total=[^ ]*'

gen_ai.request.model=gpt-5.5
gen_ai.usage.input_tokens=11
gen_ai.usage.output_tokens=16
agw.ai.usage.cost.total=0.000535

./scripts/test-bedrock.sh
./scripts/test-bedrock-lb.sh
./scripts/test-bedrock-failover.sh              # stay on Sonnet 4.5 while healthy
./scripts/test-bedrock-failover.sh failover-force
./scripts/test-bedrock-failover.sh restore

# single call (resolve the NodePort dynamically — currently 30275)
curl -sS http://172.16.10.155:30160/bedrock/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"us.anthropic.claude-sonnet-4-5-20250929-v1:0","max_tokens":16,"messages":[{"role":"user","content":"ping"}]}'

✓ bedrock       claude-sonnet-4-5   HTTP 200  · pong
✓ bedrock-lb    nova-pro / nova-lite / llama3-3-70b  HTTP 200
✓ bedrock-failover  Sonnet 4.5 → Nova Pro → Llama 3.3 70B

#Load balancing & model failover

Multi-provider resilience on one path. Same Solo patterns as the docs: P2C load balancing and priority-group failover.

./scripts/test-llm-lb-failover.sh              # LB + healthy primary
./scripts/test-llm-lb-failover.sh lb           # 8× /llm-lb — watch model field
./scripts/test-llm-lb-failover.sh failover     # stay on gpt-5.5 while healthy
./scripts/test-llm-lb-failover.sh failover-force  # force walk tiers (temp policy)
./scripts/test-llm-lb-failover.sh restore      # production health again

# AWS Bedrock LB + failover (own scripts)
./scripts/test-bedrock-lb.sh                   # P2C across Nova Pro / Nova Lite / Llama 3.3
./scripts/test-bedrock-failover.sh             # stay on Sonnet 4.5 while healthy
./scripts/test-bedrock-failover.sh failover-force  # walk Sonnet → Nova Pro → Llama
./scripts/test-bedrock-failover.sh restore     # production health again

#Workshop-gap flows Mermaid

Phases A / B / E from the FE enterprise workshop, on dedicated paths (open /openai stays free for kagent). Suite: ./scripts/test-workshop-phases.sh a|b|e · full catalog in docs/flows.md.

sequenceDiagram
  participant U as Client
  participant G as AGW /guard-demo
  participant O as OpenAI
  U->>G: clean prompt
  G->>O: forward
  O-->>U: 200
  U->>G: ignore previous instructions
  G-->>U: 403 injection
  U->>G: email or API key in prompt
  G-->>U: 422 PII or secret

./scripts/test-guardrails.sh

flowchart TB
  C[Client POST /route-by-model] --> PR[PreRouting extract body.model]
  PR --> H{x-gateway-model-name}
  H -->|gpt-5.5| O[openai]
  H -->|grok-4.5| X[xai-grok]
  H -->|gemini-2.5-flash| Ge[google-gemini]
  H -->|mock-gpt or missing| M[mock-gpt]

./scripts/test-body-routing.sh
./scripts/test-llm-token-ratelimit.sh

sequenceDiagram
  participant U as Client
  participant G as AGW /secure-openai
  participant E as Entra JWKS
  participant O as OpenAI
  U->>G: no Authorization
  G-->>U: 401
  U->>G: Bearer Entra JWT
  G->>E: validate iss/aud
  G->>O: chat/completions
  O-->>U: 200

./scripts/test-llm-jwt-entra.sh
./scripts/test-waf.sh

sequenceDiagram
  participant U as Client
  participant G as AGW /enrich-demo
  participant O as OpenAI
  Note over G: prepend system JSON contract
  U->>G: What is 2+2?
  G->>O: system + user
  O-->>U: answer + source marker

./scripts/test-prompt-enrichment.sh
./scripts/test-llm-stream.sh
./scripts/test-embeddings.sh

#Endpoints cheat-sheet