Servicer Metrics Reference¶
The Rust servicer, the worker-side process that serves an engine to the gateway, exports Prometheus families of its own, smg_servicer_*, on a listener of its own. They are not on the gateway's /metrics (see the Metrics Reference): the servicer is another process, with its own recorder, so a process that hosts a servicer beside another metrics user never shares a registry, and two servicers in one process keep two.
The listener serves GET /metrics when the servicer is given a metrics address. On the vLLM and SGLang launchers that is --metrics-port, with --metrics-host for the host, and SMG's TensorRT-LLM launcher (python -m smg_grpc_servicer.trtllm) takes the same flags; TensorRT-LLM's own trtllm-serve --grpc --grpc-protocol smg entrypoint takes no such flags, so the variables below give its address. On the TokenSpeed launcher it is --servicer-metrics-port and --servicer-metrics-host: TokenSpeed's --metrics-port and --metrics-host configure the engine's own metrics endpoint and are forwarded to the engine unchanged under either servicer implementation. Without the flags, the launcher's variables SMG_<ENGINE>_SERVICER_METRICS_PORT and SMG_<ENGINE>_SERVICER_METRICS_HOST (ENGINE the launcher's name in upper case) give the address; without either, no endpoint. When neither the flag nor the variable gives a host, the gRPC listener's host is used, or every interface when that too is unset. Port 0 binds an ephemeral port; the gRPC port itself is refused as a configuration error, and so is a metrics port with the Python servicer selected: only the Rust servicer serves the endpoint. The Rust servicer is the vLLM, SGLang and TokenSpeed launchers' default (their Python servicer is deprecated); the TensorRT-LLM launcher defaults to the engine's own Python servicer, so a TensorRT-LLM endpoint also needs the Rust servicer selected, with --servicer-impl rust or SMG_TRTLLM_SERVICER_IMPL=rust. The Rust servicer's Python classes (smg.servicer.VllmGrpcServer and its peers) take the address as metrics_bind_address (host:port). The listener binds beside the gRPC listener and before the engine connects, so a scrape during the engine's start answers; a busy port is a start-up error. Without a listener nothing is recorded beyond the token totals the servicer's own stats line reads. Histogram samples are drained into the rendered distributions every 5 seconds and at each scrape.
In the Helm chart, a gRPC workers[] entry with metrics.enabled: true enables the listener through the launcher's variables: the engine pod gets SMG_<ENGINE>_SERVICER_METRICS_PORT set to metrics.port (default 29100), and the port goes on the worker's Service, which the chart's worker ServiceMonitor scrapes under router.metrics.serviceMonitor.enabled; see Monitoring. A gRPC worker's servicerImpl value names an implementation explicitly and is exported as SMG_<ENGINE>_SERVICER_IMPL; when servicerImpl is unset, metrics.enabled exports SMG_<ENGINE>_SERVICER_IMPL=rust, naming the servicer that serves the endpoint. servicerImpl: "python" together with metrics.enabled is refused by the launcher before the engine starts.
Every family carries model (the served model) and wire (the engine family the servicer serves: vllm, sglang, tokenspeed or trtllm). The request families carry rank, the data-parallel rank the request ran on (other for a rank beyond the configured width, built on first use). Most series are registered when the servicer starts, per rank and per label value, so they are on the first scrape at zero; the exceptions are named in their entries: a gRPC code's error series at its first error, the string-stop family only on a servicer that matches stop strings itself, the media-processing families only on the vLLM wire, and the media-identity family only on the SGLang wire, its outcome series at their first count. The request path never builds a key, hashes a label or takes a lock: one clock read at arrival, one clock read plus one histogram push and two atomic adds at the first output, one atomic add (and, with the inter-token histogram on, one clock read and one push) per output chunk, and one clock read plus a handful of pushes when the stream ends.
Request Families¶
smg_servicer_requests_total¶
Requests whose stream ended, by how they ended. A fan-out (n > 1) is one request with the reason of its last choice to end, an abort winning.
| Type | Labels |
|---|---|
| Counter | rank, finish_reason |
Finish reasons: stop, length, abort, error (the engine or the wire failed the request), cut (a shutdown cut the request while it was still open). An unknown wire value counts as stop.
# Requests ending per second, by reason
sum by (finish_reason) (rate(smg_servicer_requests_total[5m]))smg_servicer_requests_refused_total¶
Requests refused before they reached the engine, by reason: duplicate_id (the id names a request still in flight), draining (the servicer is draining), invalid_argument (the request itself was invalid), media (the request's media could not be processed).
| Type | Labels |
|---|---|
| Counter | reason |
smg_servicer_requests_active¶
Admitted requests still preprocessing, streaming, or unwinding an abort.
| Type | Labels |
|---|---|
| Gauge | None |
smg_servicer_request_errors_total¶
Requests that failed with a gRPC status, by its code in lower case (internal, unavailable, cancelled, invalid_argument, ...). A code's series appears at its first error; a code that never happens renders no series.
| Type | Labels |
|---|---|
| Counter | code |
smg_servicer_prompt_tokens_total¶
Prompt tokens as the engine reported them at each request's first output.
| Type | Labels |
|---|---|
| Counter | rank |
smg_servicer_prompt_tokens_cached_total¶
Prompt tokens the engine served from its prefix cache.
| Type | Labels |
|---|---|
| Counter | rank |
smg_servicer_generation_tokens_total¶
Generated tokens streamed to callers.
| Type | Labels |
|---|---|
| Counter | rank |
# Generated tokens per second, by rank
sum by (rank) (rate(smg_servicer_generation_tokens_total[1m]))
# Prefix-cache share of the prompt tokens
sum(rate(smg_servicer_prompt_tokens_cached_total[5m])) / sum(rate(smg_servicer_prompt_tokens_total[5m]))smg_servicer_string_stops_total¶
Requests this servicer ended on a stop string it matched itself. Registered only when the servicer matches stop strings.
| Type | Labels |
|---|---|
| Counter | rank |
smg_servicer_time_to_first_token_seconds¶
Seconds from a request's arrival at the servicer to its first output. The buckets are the edges of the engines' own time-to-first-token families (1 ms to 2560 s), so a dashboard built for an engine's server reads this family by le.
| Type | Labels |
|---|---|
| Histogram | rank |
# P99 time to first token at the servicer
histogram_quantile(0.99, sum by (le) (rate(smg_servicer_time_to_first_token_seconds_bucket[5m])))smg_servicer_e2e_request_latency_seconds¶
Seconds from a request's arrival at the servicer to the end of its stream. Buckets from 0.3 s to 7680 s.
| Type | Labels |
|---|---|
| Histogram | rank |
smg_servicer_request_time_per_output_token_seconds¶
Mean seconds per generated token after the first, per request, on the servicer's clock. Buckets from 10 ms to 80 s.
| Type | Labels |
|---|---|
| Histogram | rank |
smg_servicer_inter_token_latency_seconds¶
Seconds between consecutive output chunks of a choice, per token when a chunk carries several, on the servicer's clock. Buckets from 10 ms to 80 s. The observations sit behind a pressure sampler: above a bound of pending observations the next one is dropped and counted in smg_servicer_metrics_dropped_total, so a servicer nobody scrapes cannot grow the process.
| Type | Labels |
|---|---|
| Histogram | rank |
smg_servicer_request_prompt_tokens¶
Prompt tokens per request. The token-count histograms take a 1-2-5 series of edges up to the model's context length (32768 when the engine does not report one).
| Type | Labels |
|---|---|
| Histogram | rank |
smg_servicer_request_generation_tokens¶
Generated tokens per request.
| Type | Labels |
|---|---|
| Histogram | rank |
smg_servicer_request_params_max_tokens¶
The max_tokens parameter of the requests that set it.
| Type | Labels |
|---|---|
| Histogram | rank |
smg_servicer_request_params_n¶
The n parameter of requests. Buckets 1, 2, 5, 10, 20.
| Type | Labels |
|---|---|
| Histogram | rank |
Lifecycle and Scrape Families¶
smg_servicer_state¶
The servicer's lifecycle state, one-hot: 1 on the series of the current state, 0 on the others. States: starting (the engine has not completed its handshake), serving (the engine is connected and alive, admission is open and readiness is reported), warming_up (readiness held back, the warm-up's own requests admitted only), draining (admission closed), engine_down (the engine connection failed or the engine is no longer alive).
| Type | Labels |
|---|---|
| Gauge | state |
# Servicers not serving
smg_servicer_state{state="serving"} == 0smg_servicer_start_time_seconds¶
When the servicer started, in seconds since the Unix epoch.
| Type | Labels |
|---|---|
| Gauge | None |
smg_servicer_metrics_scrapes_total¶
Scrapes answered, by HTTP status code: 200, or 500 for a scrape whose render failed (that scrape alone; the next one renders again).
| Type | Labels |
|---|---|
| Counter | code |
smg_servicer_metrics_scrape_seconds¶
How long the last scrape took to render.
| Type | Labels |
|---|---|
| Gauge | None |
smg_servicer_metrics_dropped_total¶
Observations the pressure sampler dropped, by family (inter_token_latency).
| Type | Labels |
|---|---|
| Counter | family |
Media Families¶
Requests whose media the worker side handles. The three processing families are registered only on the vLLM wire when the servicer starts, so they are on the first scrape at zero even while no request carries media. The SGLang wire records only smg_servicer_media_identity_total, whose outcome series appear at their first count; TokenSpeed and TensorRT-LLM export no media series.
smg_servicer_media_items_per_request¶
Media references a request carried, by modality (image, video, audio), counted before they are checked (a refused request counts). Buckets 1 to 64 items.
| Type | Labels |
|---|---|
| Histogram | modality |
smg_servicer_media_processing_seconds¶
Seconds a request's worker-side media spent in a stage: wait (for a processing slot) or process. The buckets are the time-to-first-token edges.
| Type | Labels |
|---|---|
| Histogram | stage |
smg_servicer_media_requests_total¶
Requests with worker-side media, by outcome: ok, refused (before processing: references this worker does not take, a missing processor, no slot), or the processor's verdict, client (the caller's media), transient (a transient failure), invalid (an invalid request), unavailable (an unavailable processor), internal (a failed processor).
| Type | Labels |
|---|---|
| Counter | outcome |
smg_servicer_media_identity_total¶
SGLang requests' worker-side media preprocessing and prefill-decode media reuse, by outcome: processed (the servicer handed a request's media references — image or video URLs without pixels — to its media processor; counted at the handoff, so a failed attempt counts), prepared (a prefill servicer built a validated pixel-less identity — token ids, placeholder ranges and grid tensors — and returned it early for its decode peer), reused (a decode servicer accepted the identity a request carried and skipped refetching and reprocessing the media), fallback (the request carried an identity that was not accepted — the servicer is not the decode side, reuse is not enabled, or validation failed — so the original inputs were kept and the media processed as usual).
Recorded only on the sglang wire. Nothing is registered up front: an outcome's series appears at its first count. One request can count under more than one outcome: a fallback that then reprocesses also counts processed, and a prefill request that preprocesses counts processed and then prepared.
| Type | Labels |
|---|---|
| Counter | outcome |