Metrics Reference¶

Complete reference for Prometheus metrics exposed by SMG. Metrics are organized in six layers matching the request lifecycle.

Metrics Endpoint¶

Metrics are exposed on the Prometheus port (default: 29000):

curl http://localhost:29000/metrics

The same listener also serves a WebSocket stream of real-time metric updates at /ws/metrics (used by the TUI and dashboards that need live state).

Configure via CLI:

smg --prometheus-port 29000 --prometheus-host 0.0.0.0

Layer 1: HTTP Metrics¶

Metrics for incoming HTTP requests at the gateway edge.

smg_http_requests_total¶

Total HTTP requests received by the gateway.

# Request rate by endpoint
sum by (path) (rate(smg_http_requests_total[5m]))

# Total request rate
sum(rate(smg_http_requests_total[5m]))

smg_http_request_duration_seconds¶

HTTP request duration from receipt to response.

# P99 latency by endpoint
histogram_quantile(0.99, sum by (path, le) (rate(smg_http_request_duration_seconds_bucket[5m])))

# Average latency
rate(smg_http_request_duration_seconds_sum[5m]) / rate(smg_http_request_duration_seconds_count[5m])

smg_http_responses_total¶

HTTP responses by status and error code.

# Error rate (5xx responses)
sum(rate(smg_http_responses_total{status_code=~"5.."}[5m])) / sum(rate(smg_http_responses_total[5m]))

# Success rate
sum(rate(smg_http_responses_total{status_code="200"}[5m])) / sum(rate(smg_http_responses_total[5m]))

smg_http_connections_active¶

Currently active HTTP connections.

smg_http_inflight_request_age_count¶

Distribution of in-flight request ages for Grafana heatmaps.

Age buckets (seconds): 30, 60, 180, 300, 600, 1200, 3600, 7200, 14400, 28800, 86400

smg_http_rate_limit_total¶

Rate limiting decisions.

Values: allowed, rejected

# Rejection rate
rate(smg_http_rate_limit_total{result="rejected"}[5m]) / sum(rate(smg_http_rate_limit_total[5m]))

Layer 2: Router Metrics¶

Metrics for request routing and processing.

smg_router_requests_total¶

Requests processed by the router.

Router types: openai, http, grpc Backend types: regular, pd, external, harmony Endpoints: chat, generate, responses, completions, rerank, embeddings, classify, messages, realtime, realtime_sessions, realtime_client_secrets, realtime_transcription Streaming: true, false

# Request rate by model
sum by (model) (rate(smg_router_requests_total[5m]))

# Streaming vs non-streaming
sum by (streaming) (rate(smg_router_requests_total[5m]))

smg_router_request_duration_seconds¶

Total router request duration.

smg_router_request_errors_total¶

Router errors by type.

Error types: no_workers, timeout, backend_error, validation_error, internal_error

# Error rate by type
sum by (error_type) (rate(smg_router_request_errors_total[5m]))

smg_router_stage_duration_seconds¶

Duration of individual pipeline stages (gRPC mode only).

Stage names are emitted by the gRPC pipeline (e.g., tokenize, route, inference, detokenize, tool_parse).

# Tokenization latency
histogram_quantile(0.99, rate(smg_router_stage_duration_seconds_bucket{stage="tokenize"}[5m]))

smg_router_ttft_seconds¶

Time to first token (gRPC streaming only).

# P50 TTFT by model
histogram_quantile(0.5, sum by (model, le) (rate(smg_router_ttft_seconds_bucket[5m])))

smg_router_tpot_seconds¶

Time per output token (gRPC streaming only).

# Average TPOT
rate(smg_router_tpot_seconds_sum[5m]) / rate(smg_router_tpot_seconds_count[5m])

smg_router_tokens_total¶

Token counts by type.

Token types: input, output

# Tokens per second
sum by (token_type) (rate(smg_router_tokens_total[5m]))

# Input/output ratio
sum(rate(smg_router_tokens_total{token_type="output"}[5m])) / sum(rate(smg_router_tokens_total{token_type="input"}[5m]))

smg_router_generation_duration_seconds¶

Total generation time (first token to last token).

smg_router_upstream_responses_total¶

HTTP responses from upstream workers.

Layer 3: Worker Metrics¶

Metrics for worker pool management and resilience.

smg_worker_pool_size¶

Number of workers in the pool.

smg_worker_connections_active¶

Active connections per worker pool.

smg_worker_requests_active¶

Active requests per worker.

# Load distribution across workers
smg_worker_requests_active / ignoring(worker) group_left sum(smg_worker_requests_active)

smg_worker_health¶

Worker health status.

# Count healthy workers
sum(smg_worker_health)

# Alert on unhealthy workers
smg_worker_health == 0

smg_worker_health_checks_total¶

Health check results.

Results: success, failure

smg_worker_selection_total¶

Worker selection events by load balancer.

smg_worker_errors_total¶

Worker-level errors by type.

Circuit Breaker Metrics¶

smg_worker_cb_state¶

Circuit breaker state per worker.

# Workers with open circuits
count(smg_worker_cb_state == 1)

smg_worker_cb_transitions_total¶

Circuit breaker state transitions.

smg_worker_cb_outcomes_total¶

Request outcomes tracked by circuit breaker.

Outcomes: success, failure

smg_worker_cb_consecutive_failures¶

Consecutive failures per worker.

smg_worker_cb_consecutive_successes¶

Consecutive successes per worker.

Retry Metrics¶

smg_worker_retries_total¶

Retry attempts.

smg_worker_retries_exhausted_total¶

Requests that exhausted all retries.

smg_worker_retry_backoff_seconds¶

Retry backoff durations by attempt number.

Layer 4: Discovery Metrics¶

Metrics for service discovery.

smg_discovery_registrations_total¶

Worker registrations.

Sources: static, kubernetes, consul, manual

smg_discovery_deregistrations_total¶

Worker deregistrations.

smg_discovery_sync_duration_seconds¶

Discovery sync duration.

smg_discovery_workers_discovered¶

Workers discovered per source.

Layer 5: MCP Tool Metrics¶

Metrics for Model Context Protocol tool execution.

smg_mcp_tool_calls_total¶

MCP tool invocations.

Results: success, error

# Tool success rate
sum(rate(smg_mcp_tool_calls_total{result="success"}[5m])) / sum(rate(smg_mcp_tool_calls_total[5m]))

# Most used tools
topk(10, sum by (tool_name) (rate(smg_mcp_tool_calls_total[5m])))

smg_mcp_tool_duration_seconds¶

Tool execution duration.

smg_mcp_servers_active¶

Active MCP servers.

smg_mcp_tool_iterations_total¶

Tool loop iterations in Responses API.

Layer 6: Database Metrics¶

Metrics for storage operations.

smg_db_operations_total¶

Database operations.

Storage types: response, conversation, conversation_item Operations: get, put, delete, list

smg_db_operation_duration_seconds¶

Database operation duration.

smg_db_connections_active¶

Active database connections.

smg_db_items_stored¶

Items stored in database.

Cache Routing Metrics¶

smg_manual_policy_cache_entries¶

Entries in the cache-aware routing cache.

smg_worker_routing_keys_active¶

Active routing keys per worker (used by cache-aware policies).

smg_manual_policy_branch_total¶

Manual policy execution branch counts for routing decisions.

smg_consistent_hashing_policy_branch_total¶

Consistent hashing policy execution branch counts for routing decisions.

smg_prefix_hash_policy_branch_total¶

Prefix hash policy execution branch counts for routing decisions.

Dashboard Queries Summary¶

Histogram Buckets¶

Default histogram buckets (29 buckets from 1ms to 7200s) applied to every metric whose name ends with duration_seconds:

0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0,
10.0, 15.0, 30.0, 45.0, 60.0, 90.0, 120.0, 180.0, 240.0, 300.0,
480.0, 900.0, 1200.0, 1800.0, 2700.0, 3600.0, 5400.0, 7200.0

Configure custom buckets via CLI:

smg --prometheus-duration-buckets 0.01 0.1 0.5 1 5 10