Shepherd Model Gateway
smg-project/smg
  • v1.9.0
  • 457
  • 136

Configure Logging

Configure structured logging with multiple output formats and integrate with log aggregation systems.

Configuration Options

SMG supports flexible logging configuration via CLI flags or environment variables.

CLI Flags

Flag Default Description
--log-level info Log level: debug, info, warn, error
--log-json false Output logs as JSON
--log-dir None Directory for log files (enables file logging)

Environment Variable

# Override log level
RUST_LOG=debug smg --worker-urls http://worker:8000

# Per-module logging
RUST_LOG=smg=debug,hyper=warn smg --worker-urls http://worker:8000

Log Levels

Level Description Use Case
error Error conditions only Production (minimal)
warn Warnings and errors Production (recommended)
info Informational messages Production (verbose)
debug Debug information Development

Set Log Level

# Via flag
smg --worker-urls http://worker:8000 --log-level debug

# Via environment
RUST_LOG=info smg --worker-urls http://worker:8000

Output Formats

Plain Text (Default)

Human-readable format with optional ANSI colors:

2024-01-15 10:30:45 INFO  smg::routing Request routed to worker worker=http://worker1:8000 policy=cache_aware

JSON Format

Machine-readable format for log aggregation:

smg --worker-urls http://worker:8000 --log-json

Output:

{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "INFO",
  "target": "smg::routing",
  "message": "Request routed to worker",
  "fields": {
    "request_id": "abc123",
    "worker": "http://worker1:8000",
    "policy": "cache_aware",
    "latency_ms": 150
  }
}

File Logging

Enable persistent log files with automatic daily rotation.

Enable File Logging

smg \
  --worker-urls http://worker:8000 \
  --log-dir /var/log/smg \
  --log-level info

Features

  • Daily rotation: New file created each day
  • Non-blocking I/O: Logging doesn't block request handling
  • File naming: smg.YYYY-MM-DD.log

Example Directory Structure

/var/log/smg/
├── smg.2024-01-13.log
├── smg.2024-01-14.log
└── smg.2024-01-15.log

Docker Logging

Basic Configuration

services:
  smg:
    image: smg:latest
    environment:
      - RUST_LOG=info
    command: >
      --worker-urls http://worker:8000
      --log-json
    logging:
      driver: json-file
      options:
        max-size: "100m"
        max-file: "3"

Production Configuration

services:
  smg:
    image: smg:latest
    command: >
      --worker-urls http://worker:8000
      --log-level info
      --log-json
      --log-dir /var/log/smg
    volumes:
      - smg-logs:/var/log/smg
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "5"

volumes:
  smg-logs:

Kubernetes Logging

Basic Pod Logging

# Follow logs
kubectl logs -n inference -l app=smg -f

# Previous container logs
kubectl logs -n inference -l app=smg --previous

# All containers in pod
kubectl logs -n inference <pod-name> --all-containers

Deployment Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: smg
spec:
  template:
    spec:
      containers:
        - name: smg
          args:
            - --worker-urls=http://worker:8000
            - --log-level=info
            - --log-json
          env:
            - name: RUST_LOG
              value: "smg=info"

Log Aggregation

Grafana Loki

server:
  http_listen_port: 9080

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: smg
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: smg
        action: keep
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
    pipeline_stages:
      - json:
          expressions:
            level: level
            target: target
            request_id: fields.request_id
      - labels:
          level:
          target:

Elasticsearch + Fluentd

<source>
  @type tail
  path /var/log/containers/smg*.log
  pos_file /var/log/fluentd-smg.pos
  tag smg.*
  <parse>
    @type json
    time_key timestamp
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

<filter smg.**>
  @type parser
  key_name log
  <parse>
    @type json
  </parse>
</filter>

<match smg.**>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name smg-logs
</match>

Log Queries

Loki/LogQL

# All SMG logs
{app="smg"}

# Error logs only
{app="smg"} | json | level="ERROR"

# Specific request
{app="smg"} | json | request_id="abc123"

# Routing decisions
{app="smg"} |= "routed" | json

# By module
{app="smg"} | json | target=~"smg::routing.*"

Elasticsearch/Kibana

{
  "query": {
    "bool": {
      "must": [
        { "match": { "target": "smg::routing" } },
        { "match": { "level": "ERROR" } }
      ],
      "filter": [
        { "range": { "timestamp": { "gte": "now-1h" } } }
      ]
    }
  }
}

OpenTelemetry Integration

When OpenTelemetry is enabled, log levels are automatically adjusted:

OTEL Enabled Event Level Behavior
Yes INFO Exported to OTLP collector
No DEBUG Not exported (local only)

This ensures request events are captured in traces when OTEL is active.

# Enable OTEL with logging
smg \
  --worker-urls http://worker:8000 \
  --enable-trace \
  --otlp-traces-endpoint localhost:4317 \
  --log-level info

Request Correlation

Request ID Propagation

SMG generates unique request IDs and propagates them through logs:

# Send request with custom ID
curl -H "X-Request-ID: my-trace-123" \
  http://localhost:30000/v1/chat/completions \
  -d '{"model": "llama", "messages": [{"role": "user", "content": "Hi"}]}'

Backend (engine) request ids derive from the same id, so gateway and worker logs correlate: gRPC requests are dispatched as {request-id}-{uuid} (the suffix keeps retries and tool-loop iterations unique), and the response body id matches. A protocol-level rid field (chat, completions, messages, generate, embeddings, classify) overrides this and is used verbatim outside PD mode.

Trace a Request

# Find all logs for a request
kubectl logs -n inference -l app=smg | grep "my-trace-123"

# In Loki
{app="smg"} | json | request_id="my-trace-123"

Log Rotation

Linux logrotate

/var/log/smg/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 smg smg
}

Docker

logging:
  driver: json-file
  options:
    max-size: "100m"
    max-file: "5"

Verification

# Check log output
smg --worker-urls http://worker:8000 --log-level debug 2>&1 | head -20

# Verify JSON format
smg --worker-urls http://worker:8000 --log-json 2>&1 | jq .

# Test log level filtering
smg --worker-urls http://worker:8000 --log-level warn 2>&1 | grep -c INFO
# Should output: 0

# Check file logging
smg --worker-urls http://worker:8000 --log-dir /tmp/smg-logs &
ls -la /tmp/smg-logs/

Troubleshooting

No logs appearing
  1. Check log level is not too restrictive:
smg --log-level debug ...
  1. Verify logs are going to stderr:
smg --worker-urls http://worker:8000 2>&1 | head
  1. Check RUST_LOG isn't overriding:
unset RUST_LOG
smg --log-level info ...
Logs not in JSON format
  1. Ensure --log-json flag is set:
smg --log-json --worker-urls http://worker:8000
  1. Verify you're reading stderr, not stdout
File logs not appearing
  1. Check directory exists and is writable:
mkdir -p /var/log/smg
chmod 755 /var/log/smg
  1. Verify --log-dir flag is set correctly
Log aggregator not receiving logs
  1. Verify JSON format is enabled
  2. Check network connectivity to aggregator
  3. Verify log format matches parser expectations

What's Next?