Monitoring¶
Set up Prometheus monitoring, OpenTelemetry tracing, and Grafana dashboards for SMG.
Enable Metrics¶
SMG always serves Prometheus metrics on a dedicated port, 29000 by default. Without --prometheus-host the listener binds the unspecified address of --host's family: :: for an IPv6 --host, 0.0.0.0 otherwise. The flags below set the listener explicitly.
Start SMG with metrics¶
smg launch \
--worker-urls http://worker:8000 \
--prometheus-port 29000 \
--prometheus-host 0.0.0.0At startup SMG logs Metrics server listening on <address> (/metrics). The port must be at least 1: --prometheus-port 0 is rejected at startup. If the port is already in use, SMG fails at startup with failed to bind metrics server on <address>.
Verify metrics endpoint¶
curl http://localhost:29000/metricsYou should see Prometheus-formatted metrics:
# HELP smg_http_requests_total Total HTTP requests by method and path
# TYPE smg_http_requests_total counter
smg_http_requests_total{method="POST",path="/v1/chat/completions"} 1234
...The engines' own metrics are available separately at /engine_metrics on the main port, aggregated from the HTTP workers' own /metrics; see Engine Metrics Passthrough.
OpenTelemetry Tracing¶
SMG supports distributed tracing via OpenTelemetry.
Enable tracing¶
smg launch \
--worker-urls http://worker:8000 \
--enable-trace \
--otlp-traces-endpoint localhost:4317Configuration¶
| Flag | Default | Description |
|---|---|---|
--enable-trace |
false |
Enable OpenTelemetry tracing |
--otlp-traces-endpoint |
localhost:4317 |
OTLP gRPC collector endpoint, as host:port (http:// is added when no scheme is given) |
Spans are batched and exported over OTLP gRPC with the service name smg. The gateway exports its own request spans: http_request for each HTTP request, including one that matches no route (method, path, request ID, status code, and latency in microseconds), and grpc_execute for each dispatch through the gRPC pipeline (request type, request ID, model, and mode). The recorded path never includes the query string, so a credential a client mistakenly puts in a query parameter stays out of spans and request logs.
The log level does not gate span export: the trace layer has its own filter, so the spans are exported even at --log-level warn and error, and under any RUST_LOG. The log level and RUST_LOG still govern what appears in the logs; see Log Levels.
Trace propagation¶
With --enable-trace, SMG uses W3C TraceContext headers:
traceparentandtracestateon an incoming client request make the gateway's spans part of the caller's trace.- Requests the HTTP routers (regular and PD) send to workers carry the current
traceparentandtracestate, and requests to gRPC workers carry the same context in their request metadata. Requests to ZMQ direct workers carry no trace context. - The response to the client carries
traceparent(plustracestatewhen the request carried one), so a client holding a reply can look up its trace.
Without --enable-trace, SMG neither reads nor adds these headers.
Prometheus Configuration¶
Basic configuration¶
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'smg'
static_configs:
- targets: ['localhost:29000']
metrics_path: /metricsTo also collect the engines' metrics through the gateway, add a job for /engine_metrics on the main port. The endpoint aggregates the HTTP workers' own /metrics, and every sample carries a worker_addr label; gRPC and ZMQ workers are not scraped there (a gRPC worker's engine pod can serve the servicer's metrics instead — see the ServiceMonitor section below):
- job_name: 'smg-engines'
static_configs:
- targets: ['localhost:30000']
metrics_path: /engine_metricsKubernetes ServiceMonitor¶
For Prometheus Operator:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: smg
namespace: inference
labels:
app: smg
spec:
selector:
matchLabels:
app: smg
endpoints:
- port: metrics
interval: 15s
path: /metrics
namespaceSelector:
matchNames:
- inferenceIf you deploy with the SMG Helm chart, set router.metrics.serviceMonitor.enabled=true and the chart creates a ServiceMonitor for the router's metrics port.
The chart can also expose worker-side metrics. A gRPC workers[] entry with metrics.enabled: true runs its engine behind the Rust servicer, and the engine pod answers GET /metrics with the servicer's own smg_servicer_* families on metrics.port (default 29100; a port equal to the worker's serving port is refused when the chart renders). An HTTP worker runs the engine's own server, which has its own /metrics; the setting is ignored there.
Under the same router.metrics.serviceMonitor.enabled flag, when at least one worker serves metrics, the chart adds a second ServiceMonitor (suffixed -worker) over the worker Services with the router ServiceMonitor's interval and labels; a worker without metrics.enabled exposes no metrics port and yields no target. The servicer's listener binds before the engine connects, so a scrape during start-up answers smg_servicer_state{state="starting"} 1.
Key Metrics by Layer¶
Layer 0: Runtime Metrics¶
| Metric | Type | Description |
|---|---|---|
smg_tokio_event_loop_stalls_total |
Counter | Event-loop stalls (async runtime blocked for more than 5 ms) |
smg_tokio_worker_busy_ratio |
Gauge | Busy fraction of each runtime worker thread |
smg_allocator_allocated_bytes |
Gauge | Bytes in live allocations (jemalloc) |
smg_allocator_resident_bytes |
Gauge | Resident bytes of the jemalloc heap |
Layer 1: HTTP Metrics¶
| Metric | Type | Description |
|---|---|---|
smg_http_requests_total |
Counter | Requests by method, path |
smg_http_request_duration_seconds |
Histogram | Request latency |
smg_http_responses_total |
Counter | Responses by path, status_code, error_code |
smg_http_connections_active |
Gauge | Requests in flight |
smg_http_rate_limit_total |
Counter | Concurrency-limit decisions |
smg_admission_queue_depth |
Gauge | Requests waiting for an admission token |
smg_admission_queue_rejected_total |
Counter | Admission rejections by reason |
Layer 2: Router Metrics¶
| Metric | Type | Description |
|---|---|---|
smg_router_requests_total |
Counter | Requests by router_type, model, endpoint |
smg_router_ttft_seconds |
Histogram | Time to first token (gRPC streaming) |
smg_router_tpot_seconds |
Histogram | Time per output token (gRPC streaming) |
smg_router_tokens_total |
Counter | Tokens by type (input/output/cached) |
smg_pd_ttft_seconds |
Histogram | End-to-end time to first token of PD requests |
smg_pd_admission_sheds_total |
Counter | PD dispatches shed because the decode engine had no room |
smg_tokenizer_cache_lookups_total |
Counter | Tokenizer cache lookups by layer, result |
Layer 3: Worker Metrics¶
| Metric | Type | Description |
|---|---|---|
smg_worker_health |
Gauge | Health status (1=Ready, 0=not Ready; a removed worker's series is dropped) |
smg_worker_requests_active |
Gauge | Active requests per worker |
smg_worker_cb_state |
Gauge | Circuit breaker state |
smg_worker_retries_total |
Counter | Retry attempts |
smg_workers_overloaded |
Gauge | Workers excluded as overloaded, per model |
smg_worker_overload_shed_total |
Counter | Requests shed with worker_overload_protection_shed, by stage |
smg_engine_token_usage |
Gauge | Engine-reported KV-cache usage per worker |
Routing Policy Metrics¶
| Metric | Type | Description |
|---|---|---|
smg_cache_tree_tokens |
Gauge | Cache-aware token-tree size per model (gRPC) |
smg_cache_tree_chars |
Gauge | Cache-aware string-tree size per model (HTTP) |
smg_cache_aware_policy_branch_total |
Counter | Cache-aware routing decisions by outcome |
Layer 5: MCP Metrics¶
| Metric | Type | Description |
|---|---|---|
smg_mcp_tool_calls_total |
Counter | Tool invocations by model, tool_name, result |
smg_mcp_tool_duration_seconds |
Histogram | Tool execution time by model, tool_name |
smg_mcp_servers_active |
Gauge | Connected MCP servers |
Several of these exist only when their feature is on: admission metrics are present unless --max-concurrent-requests is 0 or the priority scheduler is on (the multimodal rejection reasons need only --multimodal-max-inflight-bytes and are recorded either way), the smg_workers_overloaded gauge needs worker overload protection, and PD metrics need PD mode.
SMG publishes smg_worker_overload_shed_total at zero for every stage from start-up, whether or not overload protection is enabled, so the per-stage series exist before any shed; only the smg_workers_overloaded gauge waits for protection to flag a worker. See Overload Protection Metrics.
Grafana Dashboards¶
If you deploy with the SMG Helm chart, you do not have to build a dashboard by hand: set grafana.dashboard.enabled=true (default false) and the chart creates a ready-made dashboard as a ConfigMap labeled grafana_dashboard: "1", which a Grafana sidecar dashboard loader picks up. The dashboard has 49 panels in eight rows (traffic, latency, capacity and queues, workers, cache and KV events, Symphony, runtime, and prefill-decode), and its namespace and job variables scope every query to the release, so several releases can share one Grafana. The chart's README.md describes each row and panel.
For deployments without the chart, the panels below are building blocks for a dashboard of your own.
Essential panels¶
Request Rate
sum(rate(smg_http_requests_total[5m]))P99 Latency
histogram_quantile(0.99, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))Error Rate
sum(rate(smg_http_responses_total{status_code=~"5.."}[5m]))
/ sum(rate(smg_http_responses_total[5m]))/v1/responses Success Rate
sum(rate(smg_http_responses_total{path="/v1/responses",status_code=~"2.."}[5m]))
/ sum(rate(smg_http_responses_total{path="/v1/responses"}[5m]))Time to First Token (TTFT)
histogram_quantile(0.5, sum by (le) (rate(smg_router_ttft_seconds_bucket[5m])))Tokens per Second
sum(rate(smg_router_tokens_total[5m]))Healthy Workers
count(smg_worker_health == 1)Capacity and overload panels¶
Admission Queue (depth and rejections by reason)
smg_admission_queue_depth
sum by (reason) (rate(smg_admission_queue_rejected_total[5m]))Overload Shedding (sheds by stage and vetoed workers per model)
sum by (stage) (rate(smg_worker_overload_shed_total[5m]))
smg_workers_overloadedPD Admission (dispatches that waited for, or were shed by, the decode engine's running window)
rate(smg_pd_admission_waits_total[5m])
rate(smg_pd_admission_sheds_total[5m])Allocator Memory (live allocations vs. resident heap)
smg_allocator_allocated_bytes
smg_allocator_resident_bytesCache Tree Size (per model)
smg_cache_tree_tokens
smg_cache_tree_charsTokenizer Cache Hit Rate (per layer)
sum by (layer) (rate(smg_tokenizer_cache_lookups_total{result="hit"}[5m]))
/ sum by (layer) (rate(smg_tokenizer_cache_lookups_total[5m]))Event-Loop Stalls
rate(smg_tokio_event_loop_stalls_total[5m])Alerting Rules¶
The SMG Helm chart also ships alerting rules: set prometheusRule.enabled=true (default false) and the chart adds a PrometheusRule for the Prometheus Operator (its CRDs must be installed). Its eleven alerts cover gateway and worker availability, edge and router error ratios, time to first token, KV-cache capacity, circuit breakers, admission rejections, event-loop stalls, dropped log lines, and degraded KV-event ranks. Thresholds live under prometheusRule.alerts (the defaults and their rationale are in the chart's values.yaml), and prometheusRule.jobSelector scopes every expression to the release, so several releases can share one Prometheus and a firing alert names its gateway. The chart's README.md lists each alert and its firing condition.
The rules below are a starting point for deployments without the chart:
groups:
- name: smg
rules:
- alert: SMGHighErrorRate
expr: |
sum(rate(smg_http_responses_total{status_code=~"5.."}[5m]))
/ sum(rate(smg_http_responses_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate on SMG"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: SMGWorkerUnhealthy
expr: smg_worker_health == 0
for: 1m
labels:
severity: warning
annotations:
summary: "SMG worker unhealthy"
description: "Worker {{ $labels.worker }} is unhealthy"
- alert: SMGHighLatency
expr: |
histogram_quantile(0.99, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m]))) > 5
for: 5m
labels:
severity: warning
annotations:
summary: "High latency on SMG"
description: "P99 latency is {{ $value }}s"
- alert: SMGCircuitBreakerOpen
expr: smg_worker_cb_state == 1
for: 1m
labels:
severity: critical
annotations:
summary: "Circuit breaker open"
description: "Circuit breaker for {{ $labels.worker }} is open"
- alert: SMGHighTTFT
expr: |
histogram_quantile(0.95, sum by (le) (rate(smg_router_ttft_seconds_bucket[5m]))) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "High time to first token"
description: "P95 TTFT is {{ $value }}s"
- alert: SMGRateLimitRejections
expr: sum(rate(smg_http_rate_limit_total{result="rejected"}[5m])) > 10
for: 5m
labels:
severity: warning
annotations:
summary: "High rate limit rejections"
description: "{{ $value }} rejections/sec"
- alert: SMGAdmissionQueueBacklog
expr: min_over_time(smg_admission_queue_depth[10m]) > 0
labels:
severity: warning
annotations:
summary: "Admission queue has not drained for 10 minutes"
description: "At least {{ $value }} requests waiting for an admission token"
- alert: SMGOverloadShedding
expr: sum by (stage) (rate(smg_worker_overload_shed_total[5m])) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "SMG is shedding requests at stage {{ $labels.stage }}"
description: "{{ $value }} requests/sec answered with 503 worker_overload_protection_shed"
- alert: SMGEventLoopStalls
expr: rate(smg_tokio_event_loop_stalls_total[5m]) > 1
for: 10m
labels:
severity: warning
annotations:
summary: "SMG async runtime is stalling"
description: "{{ $value }} event-loop stalls/sec"SMGOverloadShedding also covers PD admission, whose sheds are counted with stage="pd_admission". Tune the thresholds to your traffic.
Useful Queries¶
Request analysis¶
# Request rate by endpoint
sum by (path) (rate(smg_http_requests_total[5m]))
# Success rate
sum(rate(smg_http_responses_total{status_code="200"}[5m]))
/ sum(rate(smg_http_responses_total[5m]))
# Latency percentiles
histogram_quantile(0.50, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.95, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))
# Gateway-generated errors (sheds, rejections) by code
sum by (error_code) (rate(smg_http_responses_total{error_code!=""}[5m]))LLM performance¶
# Tokens per second by model
sum by (model) (rate(smg_router_tokens_total[5m]))
# TTFT by model
histogram_quantile(0.5, sum by (model, le) (rate(smg_router_ttft_seconds_bucket[5m])))
# Input/output token ratio
sum(rate(smg_router_tokens_total{token_type="output"}[5m]))
/ sum(rate(smg_router_tokens_total{token_type="input"}[5m]))Worker analysis¶
# Load distribution
smg_worker_requests_active / ignoring(worker) group_left sum without (worker) (smg_worker_requests_active)
# Unhealthy workers
count(smg_worker_health == 0)
# Circuit breaker states
count by (worker) (smg_worker_cb_state == 1)
# Engine KV-cache usage per worker (non-Ready workers report -1; a removed worker's series is dropped)
max by (worker) (smg_engine_token_usage >= 0)MCP tool analysis¶
# Tool success rate
sum(rate(smg_mcp_tool_calls_total{result="success"}[5m]))
/ sum(rate(smg_mcp_tool_calls_total[5m]))
# Most used tools
topk(10, sum by (tool_name) (rate(smg_mcp_tool_calls_total[5m])))
# Slowest tools
topk(5, histogram_quantile(0.95, sum by (tool_name, le) (rate(smg_mcp_tool_duration_seconds_bucket[5m]))))Verification¶
# Check metrics are being scraped
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="smg")'
# Query a metric
curl -s 'http://prometheus:9090/api/v1/query?query=smg_http_requests_total' | jq
# Check alerts
curl -s http://prometheus:9090/api/v1/alerts | jq
# Engine load the gateway is routing on (cached snapshot, fleet totals)
curl -s http://localhost:30000/loads | jq '.aggregate'Troubleshooting¶
Metrics endpoint not responding
Check the startup log for
Metrics server listening on <address> (/metrics), which shows the address actually bound. Afailed to bind metrics servererror usually means the port is already in use.Check the port is listening:
netstat -tlnp | grep 29000- Check firewall rules allow access
Traces not appearing
- Verify the OTLP endpoint is reachable from the gateway:
nc -zv localhost 4317Check SMG was started with
--enable-traceVerify the collector is receiving spans. Requests to HTTP and gRPC workers carry trace context; requests to ZMQ direct workers do not, so their engine-side spans cannot join the trace.
The log level does not gate span export: the spans are exported even at
--log-level warnanderror, and under anyRUST_LOG. The level still governs the log output, so log lines missing span fields such asrequest_iddo not mean tracing is off.
Missing metrics
Most series appear only after the first event that records them, so a metric can be absent on an idle gateway. The overload-shed, retry, and circuit-breaker transition families are exceptions, published at zero before any event (details below).
Some groups need their feature enabled: admission metrics are present unless
--max-concurrent-requestsis0or the priority scheduler is on, thesmg_workers_overloadedgauge needs worker overload protection, and PD, discovery, RL, and mesh metrics need their modes. Allocator metrics are absent on musl and MSVC builds.Some metrics only appear for specific paths (for example, TTFT is recorded for streaming responses on the gRPC router).
Verify metric names against the Metrics Reference. Names found on older dashboards, such as
smg_router_stage_duration_secondsand thesmg_db_*metrics, do not exist in SMG and are never emitted.
SMG publishes the overload-shed and retry families at zero from start-up, whether or not overload protection is enabled. Each worker's circuit-breaker transition counters are likewise published at zero when that worker is registered. The start-up retry series cover the regular, prefill, and decode worker types only; the external-provider counters (worker_type="external") appear only on their first event: smg_worker_retries_total on the first retry, smg_worker_retries_exhausted_total on the first exhaustion. Absence of the start-up families therefore points at a scrape problem rather than inactivity, while a missing transition series means no worker with that label is registered. The Metrics Reference lists the exact series.
What's Next?¶
- Configure Logging — Structured log aggregation
- Configure TLS — Secure client-to-gateway traffic
- Metrics Reference — Complete metrics documentation