Monitoring

Set up Prometheus monitoring, OpenTelemetry tracing, and Grafana dashboards for SMG.


Enable Metrics

SMG always serves Prometheus metrics on a dedicated port, 29000 by default. Without --prometheus-host the listener binds the unspecified address of --host's family: :: for an IPv6 --host, 0.0.0.0 otherwise. The flags below set the listener explicitly.

Start SMG with metrics

smg launch \
  --worker-urls http://worker:8000 \
  --prometheus-port 29000 \
  --prometheus-host 0.0.0.0

At startup SMG logs Metrics server listening on <address> (/metrics). The port must be at least 1: --prometheus-port 0 is rejected at startup. If the port is already in use, SMG fails at startup with failed to bind metrics server on <address>.

Verify metrics endpoint

curl http://localhost:29000/metrics

You should see Prometheus-formatted metrics:

# HELP smg_http_requests_total Total HTTP requests by method and path
# TYPE smg_http_requests_total counter
smg_http_requests_total{method="POST",path="/v1/chat/completions"} 1234
...

The engines' own metrics are available separately at /engine_metrics on the main port; see Engine Metrics Passthrough.


OpenTelemetry Tracing

SMG supports distributed tracing via OpenTelemetry.

Enable tracing

smg launch \
  --worker-urls http://worker:8000 \
  --enable-trace \
  --otlp-traces-endpoint localhost:4317

Configuration

Flag Default Description
--enable-trace false Enable OpenTelemetry tracing
--otlp-traces-endpoint localhost:4317 OTLP gRPC collector endpoint, as host:port (http:// is added when no scheme is given)

Spans are batched and exported over OTLP gRPC with the service name smg. The gateway exports its own request spans: http_request for each HTTP request (method, path, request ID, status code, and latency in microseconds) and grpc_execute for each dispatch through the gRPC pipeline (request type, request ID, model, and mode). The recorded path never includes the query string, so a credential a client mistakenly puts in a query parameter stays out of spans and request logs.

Trace propagation

With --enable-trace, SMG uses W3C TraceContext headers:

  • traceparent and tracestate on an incoming client request make the gateway's spans part of the caller's trace.
  • Requests the HTTP routers (regular and PD) send to workers carry the current traceparent and tracestate, and requests to gRPC workers carry the same context in their request metadata. Requests to ZMQ direct workers carry no trace context.
  • The response to the client carries traceparent (plus tracestate when the request carried one), so a client holding a reply can look up its trace.

Without --enable-trace, SMG neither reads nor adds these headers.


Prometheus Configuration

Basic configuration

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'smg'
    static_configs:
      - targets: ['localhost:29000']
    metrics_path: /metrics

To also collect the engines' metrics through the gateway, add a job for /engine_metrics on the main port. Every sample carries a worker_addr label:

  - job_name: 'smg-engines'
    static_configs:
      - targets: ['localhost:30000']
    metrics_path: /engine_metrics

Kubernetes ServiceMonitor

For Prometheus Operator:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: smg
  namespace: inference
  labels:
    app: smg
spec:
  selector:
    matchLabels:
      app: smg
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics
  namespaceSelector:
    matchNames:
      - inference

If you deploy with the SMG Helm chart, set router.metrics.serviceMonitor.enabled=true and the chart creates a ServiceMonitor for the router's metrics port.

The chart can also expose worker-side metrics. A gRPC workers[] entry with metrics.enabled: true runs its engine behind the Rust servicer, and the engine pod answers GET /metrics with the servicer's own smg_servicer_* families on metrics.port (default 29100; a port equal to the worker's serving port is refused when the chart renders). An HTTP worker runs the engine's own server, which has its own /metrics; the setting is ignored there.

Under the same router.metrics.serviceMonitor.enabled flag, when at least one worker serves metrics, the chart adds a second ServiceMonitor (suffixed -worker) over the worker Services with the router ServiceMonitor's interval and labels; a worker without metrics.enabled exposes no metrics port and yields no target. The servicer's listener binds before the engine connects, so a scrape during start-up answers smg_servicer_state{state="starting"} 1.


Key Metrics by Layer

Layer 0: Runtime Metrics

Metric Type Description
smg_tokio_event_loop_stalls_total Counter Event-loop stalls (async runtime blocked for more than 5 ms)
smg_tokio_worker_busy_ratio Gauge Busy fraction of each runtime worker thread
smg_allocator_allocated_bytes Gauge Bytes in live allocations (jemalloc)
smg_allocator_resident_bytes Gauge Resident bytes of the jemalloc heap

Layer 1: HTTP Metrics

Metric Type Description
smg_http_requests_total Counter Requests by method, path
smg_http_request_duration_seconds Histogram Request latency
smg_http_responses_total Counter Responses by path, status_code, error_code
smg_http_connections_active Gauge Requests in flight
smg_http_rate_limit_total Counter Concurrency-limit decisions
smg_admission_queue_depth Gauge Requests waiting for an admission token
smg_admission_queue_rejected_total Counter Admission rejections by reason

Layer 2: Router Metrics

Metric Type Description
smg_router_requests_total Counter Requests by router_type, model, endpoint
smg_router_ttft_seconds Histogram Time to first token (gRPC streaming)
smg_router_tpot_seconds Histogram Time per output token (gRPC streaming)
smg_router_tokens_total Counter Tokens by type (input/output/cached)
smg_pd_ttft_seconds Histogram End-to-end time to first token of PD requests
smg_pd_admission_sheds_total Counter PD dispatches shed because the decode engine had no room
smg_tokenizer_cache_lookups_total Counter Tokenizer cache lookups by layer, result

Layer 3: Worker Metrics

Metric Type Description
smg_worker_health Gauge Health status (1=Ready, 0=not Ready); a removed worker's series is dropped
smg_worker_requests_active Gauge Active requests per worker
smg_worker_cb_state Gauge Circuit breaker state
smg_worker_retries_total Counter Retry attempts
smg_workers_overloaded Gauge Workers excluded as overloaded, per model
smg_worker_overload_shed_total Counter Requests shed with worker_overload_protection_shed, by stage
smg_engine_token_usage Gauge Engine-reported KV-cache usage per worker

Routing Policy Metrics

Metric Type Description
smg_cache_tree_tokens Gauge Cache-aware token-tree size per model (gRPC)
smg_cache_tree_chars Gauge Cache-aware string-tree size per model (HTTP)
smg_cache_aware_policy_branch_total Counter Cache-aware routing decisions by outcome

Layer 5: MCP Metrics

Metric Type Description
smg_mcp_tool_calls_total Counter Tool invocations by model, tool_name, result
smg_mcp_tool_duration_seconds Histogram Tool execution time by model, tool_name
smg_mcp_servers_active Gauge Connected MCP servers

Several of these exist only when their feature is on: admission metrics are present unless --max-concurrent-requests is 0 (the multimodal rejection reasons need --multimodal-max-inflight-bytes), the smg_workers_overloaded gauge needs worker overload protection, and PD metrics need PD mode.

SMG publishes smg_worker_overload_shed_total at zero for every stage from start-up, whether or not overload protection is enabled, so the per-stage series exist before any shed; only the smg_workers_overloaded gauge waits for protection to flag a worker. See Overload Protection Metrics.

View all metrics →


Grafana Dashboards

If you deploy with the SMG Helm chart, grafana.dashboard.enabled=true ships a ready-made dashboard over these families as a ConfigMap, labelled grafana_dashboard: "1" for Grafana's dashboard sidecar (override the labels with grafana.dashboard.labels).

Essential panels

Request Rate

sum(rate(smg_http_requests_total[5m]))

P99 Latency

histogram_quantile(0.99, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))

Error Rate

sum(rate(smg_http_responses_total{status_code=~"5.."}[5m]))
/ sum(rate(smg_http_responses_total[5m]))

/v1/responses Success Rate

sum(rate(smg_http_responses_total{path="/v1/responses",status_code=~"2.."}[5m]))
/ sum(rate(smg_http_responses_total{path="/v1/responses"}[5m]))

Time to First Token (TTFT)

histogram_quantile(0.5, sum by (le) (rate(smg_router_ttft_seconds_bucket[5m])))

Tokens per Second

sum(rate(smg_router_tokens_total{token_type!="cached"}[5m]))

The cached type counts prompt tokens the engine served from its cache — a share of input counted again — so leave it out of throughput sums.

Healthy Workers

count(smg_worker_health == 1)

Capacity and overload panels

Admission Queue (depth and rejections by reason)

smg_admission_queue_depth
sum by (reason) (rate(smg_admission_queue_rejected_total[5m]))

Overload Shedding (sheds by stage and vetoed workers per model)

sum by (stage) (rate(smg_worker_overload_shed_total[5m]))
smg_workers_overloaded

PD Admission (dispatches that waited for, or were shed by, the decode engine's running window)

rate(smg_pd_admission_waits_total[5m])
rate(smg_pd_admission_sheds_total[5m])

Allocator Memory (live allocations vs. resident heap)

smg_allocator_allocated_bytes
smg_allocator_resident_bytes

Cache Tree Size (per model)

smg_cache_tree_tokens
smg_cache_tree_chars

Tokenizer Cache Hit Rate (per layer)

sum by (layer) (rate(smg_tokenizer_cache_lookups_total{result="hit"}[5m]))
/ sum by (layer) (rate(smg_tokenizer_cache_lookups_total[5m]))

Event-Loop Stalls

rate(smg_tokio_event_loop_stalls_total[5m])

Alerting Rules

groups:
  - name: smg
    rules:
      - alert: SMGHighErrorRate
        expr: |
          sum(rate(smg_http_responses_total{status_code=~"5.."}[5m]))
          / sum(rate(smg_http_responses_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on SMG"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: SMGWorkerUnhealthy
        expr: smg_worker_health == 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "SMG worker unhealthy"
          description: "Worker {{ $labels.worker }} is unhealthy"

      - alert: SMGHighLatency
        expr: |
          histogram_quantile(0.99, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m]))) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency on SMG"
          description: "P99 latency is {{ $value }}s"

      - alert: SMGCircuitBreakerOpen
        expr: smg_worker_cb_state == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Circuit breaker open"
          description: "Circuit breaker for {{ $labels.worker }} is open"

      - alert: SMGHighTTFT
        expr: |
          histogram_quantile(0.95, sum by (le) (rate(smg_router_ttft_seconds_bucket[5m]))) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High time to first token"
          description: "P95 TTFT is {{ $value }}s"

      - alert: SMGRateLimitRejections
        expr: sum(rate(smg_http_rate_limit_total{result="rejected"}[5m])) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High rate limit rejections"
          description: "{{ $value }} rejections/sec"

      - alert: SMGAdmissionQueueBacklog
        expr: min_over_time(smg_admission_queue_depth[10m]) > 0
        labels:
          severity: warning
        annotations:
          summary: "Admission queue has not drained for 10 minutes"
          description: "At least {{ $value }} requests waiting for an admission token"

      - alert: SMGOverloadShedding
        expr: sum by (stage) (rate(smg_worker_overload_shed_total[5m])) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "SMG is shedding requests at stage {{ $labels.stage }}"
          description: "{{ $value }} requests/sec answered with 503 worker_overload_protection_shed"

      - alert: SMGEventLoopStalls
        expr: rate(smg_tokio_event_loop_stalls_total[5m]) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "SMG async runtime is stalling"
          description: "{{ $value }} event-loop stalls/sec"

SMGOverloadShedding also covers PD admission, whose sheds are counted with stage="pd_admission". Tune the thresholds to your traffic.

The Helm chart can install alerts for you instead: prometheusRule.enabled=true renders a PrometheusRule with ten serving alerts over these families, scoped to the release by namespace and job; the thresholds sit under prometheusRule.alerts.


Useful Queries

Request analysis

# Request rate by endpoint
sum by (path) (rate(smg_http_requests_total[5m]))

# Success rate
sum(rate(smg_http_responses_total{status_code="200"}[5m]))
/ sum(rate(smg_http_responses_total[5m]))

# Latency percentiles
histogram_quantile(0.50, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.95, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (le) (rate(smg_http_request_duration_seconds_bucket[5m])))

# Gateway-generated errors (sheds, rejections) by code
sum by (error_code) (rate(smg_http_responses_total{error_code!=""}[5m]))

LLM performance

# Tokens per second by model
sum by (model) (rate(smg_router_tokens_total{token_type!="cached"}[5m]))

# TTFT by model
histogram_quantile(0.5, sum by (model, le) (rate(smg_router_ttft_seconds_bucket[5m])))

# Input/output token ratio
sum(rate(smg_router_tokens_total{token_type="output"}[5m]))
/ sum(rate(smg_router_tokens_total{token_type="input"}[5m]))

Worker analysis

# Load distribution
smg_worker_requests_active / ignoring(worker) group_left sum without (worker) (smg_worker_requests_active)

# Unhealthy workers
count(smg_worker_health == 0)

# Circuit breaker states
count by (worker) (smg_worker_cb_state == 1)

# Engine KV-cache usage per worker (a non-Ready worker reports -1; a removed worker's series is dropped)
max by (worker) (smg_engine_token_usage >= 0)

MCP tool analysis

# Tool success rate
sum(rate(smg_mcp_tool_calls_total{result="success"}[5m]))
/ sum(rate(smg_mcp_tool_calls_total[5m]))

# Most used tools
topk(10, sum by (tool_name) (rate(smg_mcp_tool_calls_total[5m])))

# Slowest tools
topk(5, histogram_quantile(0.95, sum by (tool_name, le) (rate(smg_mcp_tool_duration_seconds_bucket[5m]))))

Verification

# Check metrics are being scraped
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.job=="smg")'

# Query a metric
curl -s 'http://prometheus:9090/api/v1/query?query=smg_http_requests_total' | jq

# Check alerts
curl -s http://prometheus:9090/api/v1/alerts | jq

# Engine load the gateway is routing on (cached snapshot, fleet totals)
curl -s http://localhost:30000/loads | jq '.aggregate'

Troubleshooting

Metrics endpoint not responding
  1. Check the startup log for Metrics server listening on <address> (/metrics), which shows the address actually bound. A failed to bind metrics server error usually means the port is already in use.

  2. Check the port is listening:

netstat -tlnp | grep 29000
  1. Check firewall rules allow access
Traces not appearing
  1. Verify the OTLP endpoint is reachable from the gateway:
nc -zv localhost 4317
  1. Check SMG was started with --enable-trace

  2. Verify the collector is receiving spans. Requests to HTTP and gRPC workers carry trace context; requests to ZMQ direct workers do not, so their engine-side spans cannot join the trace.

  3. The log level is not the cause: the trace layer keeps its own filter, so the request spans are exported under --log-level warn or error too.

Missing metrics
  1. Most series appear only after the first event that records them, so a metric can be absent on an idle gateway. The overload-shed and retry families and each worker's request and circuit-breaker transition counters are exceptions, published at zero before any event (details below).

  2. Some groups need their feature enabled: admission metrics are present unless --max-concurrent-requests is 0, the smg_workers_overloaded gauge needs worker overload protection, and PD, discovery, RL, and mesh metrics need their modes. Allocator metrics are absent on musl and MSVC builds.

  3. Some metrics only appear for specific paths (for example, TTFT is recorded for streaming responses on the gRPC router).

  4. Verify metric names against the Metrics Reference. Names found on older dashboards, such as smg_router_stage_duration_seconds and the smg_db_* metrics, do not exist in SMG and are never emitted.

SMG publishes the overload-shed and retry families at zero from start-up, whether or not overload protection is enabled. Each worker's request counter (smg_worker_requests_total) and circuit-breaker transition counters are likewise published at zero when that worker is registered, so a worker that receives no traffic shows a zero request count instead of no series. The start-up retry series cover the regular, prefill, and decode worker types only; the external-provider counters (worker_type="external") appear only on their first event: smg_worker_retries_total on the first retry, smg_worker_retries_exhausted_total on the first exhaustion. Absence of the start-up families therefore points at a scrape problem rather than inactivity, while a missing per-worker request or transition series means no worker with that label is registered. The Metrics Reference lists the exact series.


What's Next?