Configure TLS

This guide shows how to serve SMG over HTTPS, and how to present a client certificate to workers that require mutual TLS (mTLS).


Overview

SMG supports TLS configurations for securing communications:

Configuration Purpose Status
Server TLS HTTPS for client → gateway communication Available
Client mTLS Mutual TLS for gateway → HTTP worker communication Available
gRPC worker transport Gateway → gRPC worker (grpc://) connections Plaintext; mTLS through a service mesh
Mesh mTLS Mutual TLS for router → router communication in the high-availability mesh Available (--mesh-tls-* flags)

Generate Certificates

For testing, generate self-signed certificates:

Step 1: Create CA

# Generate CA private key
openssl genrsa -out ca.key 4096

# Generate CA certificate
openssl req -new -x509 -days 365 -key ca.key -out ca.crt \
  -subj "/CN=SMG CA/O=SMG"

Step 2: Create server certificate

# Generate server private key
openssl genrsa -out server.key 2048

# Generate server CSR
openssl req -new -key server.key -out server.csr \
  -subj "/CN=smg.example.com/O=SMG"

# Set the DNS name that HTTPS clients will verify
cat > server.ext <<'EOF'
subjectAltName = DNS:smg.example.com
EOF

# Sign with CA and include the DNS subject alternative name (SAN)
openssl x509 -req -days 365 -in server.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -extfile server.ext -out server.crt

Step 3: Create client certificate (for mTLS to workers)

# Generate client private key
openssl genrsa -out client.key 2048

# Generate client CSR
openssl req -new -key client.key -out client.csr \
  -subj "/CN=smg-client/O=SMG"

# Sign with CA
openssl x509 -req -days 365 -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt

Enable Server TLS

Serve the gateway over HTTPS.

Configuration

smg \
  --worker-urls http://worker:8000 \
  --tls-cert-path /path/to/server.crt \
  --tls-key-path /path/to/server.key \
  --host 0.0.0.0 \
  --port 8443

Set both flags or neither: SMG refuses to start with only one of them, or when it can't read one of the files. Both files are PEM. The example serves on 8443: the published image runs as uid 65532 with every capability dropped by the Helm chart, so it cannot bind a port below 1024; put a Service, an ingress or a load balancer in front for 443.

HTTPS covers the data listener (--port); when --admin-port puts the control plane on an admin listener of its own, that listener presents the same certificate. The metrics listener (--prometheus-port) and the probe listener (--health-check-port, see Gateway Probe Endpoints) stay plain HTTP.

The TLS policy is fixed: TLS 1.2 and 1.3 only (a client limited to TLS 1.1 or older cannot connect), with rustls' default cipher suites, and no flag for a minimum version, a cipher list or SNI-based certificate selection. The certificate and key are read once at start-up, so a renewed certificate takes effect at the next restart. Plain HTTP sent to the TLS port is refused at the handshake.

At startup SMG also checks the Unix mode of each TLS private key file it was given: the server key (--tls-key-path), the client key (--client-key-path), and the mesh key (--mesh-tls-key, see Mesh mTLS). A key owned by the user the gateway runs as should be 0600 or 0400: any permission bit for its group or for others triggers a warning. A key another user owns — a secret volume's key, or a root:ssl-cert key — may also be group-readable (0640), since group read is how the gateway reads it at all; group write, or any permission for others, still warns. The warning is one log line per key, naming the file, the flag and the mode — never the key's content. It does not block startup: a secret manager may tighten the mode after the file lands.

Verification

curl --cacert ca.crt https://smg.example.com/health

Client mTLS to Workers

When your workers serve HTTPS and verify client certificates, give the gateway a client certificate and its key, plus the CA that signed the workers' server certificates:

smg \
  --worker-urls https://worker1:8443 https://worker2:8443 \
  --client-cert-path /path/to/client.crt \
  --client-key-path /path/to/client.key \
  --ca-cert-paths /path/to/ca.crt
Flag Description
--client-cert-path Client certificate (PEM) that the gateway presents to workers. Set it together with --client-key-path.
--client-key-path Private key (PEM) for the client certificate.
--ca-cert-paths One or more CA certificates (PEM) that the gateway trusts, in addition to the system's trusted roots, when it verifies worker certificates. Takes several paths after one flag, and the flag can repeat.
  • The flags are in both the Rust smg binary and the Python launcher (smg launch from pip, and the container image).
  • SMG reads the files at startup. Setting only one of --client-cert-path and --client-key-path, or passing a file it can't read, stops startup.
  • The startup permission check described in Enable Server TLS covers the client key too: an over-permissive mode logs one warning, and the gateway still starts.
  • The certificate and CAs apply to the gateway's HTTPS connections only: the per-worker clients that send requests, health probes and admin calls to HTTP workers, and the shared client for external providers and model discovery. gRPC workers (grpc://, grpcs://) don't use them; see Gateway to gRPC Workers.

Gateway to gRPC Workers

The connection from the gateway to a gRPC worker (a grpc:// URL, or a pod that service discovery registered) is plaintext, and neither end can be configured otherwise:

  • The gateway builds every gRPC channel with the same keep-alive and window profile and attaches no TLS configuration to it: no server-certificate verification, no CA bundle and no client certificate. --client-cert-path, --client-key-path and --ca-cert-paths act on the HTTPS clients above and change nothing on a gRPC dial. A grpcs:// URL is accepted, but the dial fails at once: the log shows grpc health check failed for explicitly configured worker URL grpcs://...: gRPC connection failed: transport error, the registration is retried, and the worker never registers.
  • The vLLM gRPC entrypoint, and the TokenSpeed and MLX servicers, listen on an insecure port only. The SGLang servicer accepts --ssl-certfile and --ssl-keyfile and, when both are set, serves TLS on its port. It also accepts --ssl-ca-certs and reads the CA file (re-reading it under --enable-ssl-refresh), but the servicer never requests a client certificate, so the flag does not enable client-certificate authentication: a client without a certificate still connects. That hop is not mutually authenticated either.

SMG has no native TLS on the gRPC channel. The supported way to encrypt and authenticate this hop is a service mesh: run the gateway and the workers with sidecar proxies (for example Istio or Linkerd) in strict mutual-TLS mode. The gateway keeps dialing grpc:// to its local proxy, and the proxies secure the connection between the pods. What to check in the mesh configuration:

  • HTTP/2 passthrough. The proxies must treat the worker port as gRPC (HTTP/2), for example through a Service port named grpc-<name> or appProtocol: grpc, so the long-lived streams are proxied as streams rather than buffered: one Generate stream per in-flight request, and by default one SubscribeKvEvents stream per gRPC worker, open for as long as the worker is registered. Under a cache-aware routing policy that stream carries the KV cache events and the worker's load records; under any other policy it carries the load records alone. With --disable-load-monitoring it opens only for a worker whose loads something still needs, such as a load-aware policy or overload protection.
  • Pod-IP destinations. Service discovery registers each pod as grpc://<pod-ip>:<port>, not by Service name. Confirm that the mesh applies mutual TLS to connections addressed directly to the pod IPs of workloads in the mesh, or pass --worker-urls with names the mesh routes.
  • Keepalive and idle timeouts. The gateway sends an HTTP/2 keepalive ping every 30 seconds on each channel and expects the answer within 10 seconds; a connect attempt is bounded at 10 seconds. The local proxy answers the pings itself, so a worker that dies behind a live proxy is noticed by failing requests and by the health probe (every --health-check-interval-secs seconds, 60 by default, after --health-failure-threshold consecutive failures, 3 by default), not by the keepalive. On the SubscribeKvEvents stream the servicers — the Rust servicer and the Python servicers alike — send a load-only heartbeat after one second of silence, and every five seconds on an idle engine, so a proxy's per-stream idle timeout (Envoy's default is five minutes) does not reset it. A subscription that is reset anyway is reconnected with backoff and resumed from its last sequence.

Full TLS Configuration

One command can serve HTTPS on the gateway's own listeners and present a client certificate to HTTPS workers:

smg \
  --worker-urls https://worker1:8443 https://worker2:8443 \
  --tls-cert-path /etc/certs/server.crt \
  --tls-key-path /etc/certs/server.key \
  --client-cert-path /etc/certs/client.crt \
  --client-key-path /etc/certs/client.key \
  --ca-cert-paths /etc/certs/ca.crt \
  --api-key "${API_KEY}" \
  --host 0.0.0.0 \
  --port 443

The same flags work with the Python launcher (smg launch from pip, and the container image).


Kubernetes with cert-manager

Use cert-manager for automatic certificate management.

Step 1: Install cert-manager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml

Step 2: Create Certificate

Before continuing, create the inference namespace and configure a ClusterIssuer named letsencrypt-prod for your domain and ACME challenge method. Installing cert-manager does not create this issuer. Follow the cert-manager issuer configuration guide and confirm the issuer is ready:

kubectl wait --for=condition=Ready clusterissuer/letsencrypt-prod --timeout=120s
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: smg-tls
  namespace: inference
spec:
  secretName: smg-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - smg.example.com

Apply the certificate and wait for issuance before mounting its secret:

kubectl apply -f smg-certificate.yaml
kubectl wait --for=condition=Ready certificate/smg-tls -n inference --timeout=300s

Step 3: Mount in deployment

spec:
  containers:
    - name: smg
      volumeMounts:
        - name: tls-certs
          mountPath: /etc/certs
          readOnly: true
      args:
        - --tls-cert-path
        - /etc/certs/tls.crt
        - --tls-key-path
        - /etc/certs/tls.key
  volumes:
    - name: tls-certs
      secret:
        secretName: smg-tls-secret

Verification

Test server TLS

# With CA certificate
curl --cacert ca.crt https://smg.example.com/health

# Check certificate details
openssl s_client -connect smg.example.com:443 -showcerts

Test worker connectivity

# Check SMG logs for worker connections
kubectl logs -n inference -l app=smg | grep -i worker

# Verify worker connection via control plane API
curl --cacert ca.crt https://smg.example.com/workers

Troubleshooting

Certificate verification failed
  1. Verify CA certificate matches:
openssl verify -CAfile ca.crt server.crt
  1. Check certificate expiration:
openssl x509 -in server.crt -noout -dates
  1. Verify hostname matches:
openssl x509 -in server.crt -noout -text | grep DNS
Connection refused
  1. Check SMG is listening on correct port:
netstat -tlnp | grep smg
  1. Verify TLS configuration in logs:
smg --tls-cert-path ... 2>&1 | grep -i tls
Handshake failure
  1. Check TLS version compatibility
  2. Verify cipher suite support
  3. Ensure certificate chain is complete

What's Next?