Data Connections¶
This guide helps you quickly enable conversation history storage with a backend that matches your environment.
Choose a Backend¶
SMG supports these history backends via --history-backend:
memory(default): in-process, non-persistent, with bounded retentionnone: disable history storagepostgres: durable relational storageredis: fast key-value storage with optional retentionoracle: enterprise Oracle backend
Quick Start Commands¶
Memory (default)¶
smg \
--worker-urls http://worker:8000 \
--history-backend memoryThe memory backend is bounded: on every write it drops the oldest records to keep at most 10,000 stored responses and 100,000 conversation items, and drops records older than 24 hours (a conversation's age counts from its last use). Change these bounds with --history-memory-max-responses, --history-memory-max-items, and --history-memory-retention-secs (a retention of 0 keeps a record until a count bound drops it). See the memory backend options for details.
No history¶
smg \
--worker-urls http://worker:8000 \
--history-backend nonePostgreSQL¶
smg \
--worker-urls http://worker:8000 \
--history-backend postgres \
--postgres-db-url "postgres://user:password@localhost:5432/smg" \
--postgres-pool-max-size 16A new, empty database needs nothing more: the first start creates the tables at the latest schema version. An existing database that is behind a newer SMG build has pending migrations to apply; see Schema migrations.
To keep the password off the command line, set SMG_POSTGRES_DB_URL or pass --postgres-db-url-file with a file holding the URL instead; see Store URLs from the environment or a file.
Redis¶
smg \
--worker-urls http://worker:8000 \
--history-backend redis \
--redis-url "redis://localhost:6379" \
--redis-pool-max-size 16 \
--redis-retention-days 30Set --redis-retention-days -1 for persistent retention.
To keep a password off the command line, set SMG_REDIS_URL or pass --redis-url-file with a file holding the URL instead; see Store URLs from the environment or a file.
Oracle¶
smg \
--worker-urls http://worker:8000 \
--history-backend oracle \
--oracle-wallet-path /path/to/wallet \
--oracle-tns-alias mydb_high \
--oracle-user admin \
--oracle-password "$ORACLE_PASSWORD"On a new database, also set DB_AUTO_MIGRATE=true for the first start; see Schema migrations.
Schema migrations¶
PostgreSQL and Oracle track a schema version and refuse to start while migrations are pending, printing the SQL to apply by hand.
A new, empty PostgreSQL database has nothing pending: the first start creates its tables at the latest schema version, and gateways starting together on one database take turns at schema setup and all come up.
A new Oracle database starts with every migration pending, and an existing database on either backend can fall behind after an SMG upgrade. Set DB_AUTO_MIGRATE=true (or 1) to let SMG apply pending migrations at startup:
DB_AUTO_MIGRATE=true smg launch \
--worker-urls http://worker:8000 \
--history-backend postgres \
--postgres-db-url "postgres://user:password@localhost:5432/smg"A --schema-config file can set auto_migrate instead. See Chat History.
Required Settings by Backend¶
| Backend | Required settings |
|---|---|
memory |
none |
none |
none |
postgres |
the PostgreSQL connection URL, via --postgres-db-url, SMG_POSTGRES_DB_URL, or a URL file |
redis |
the Redis connection URL, via --redis-url, SMG_REDIS_URL, or a URL file |
oracle |
--oracle-user, --oracle-password, and one of (--oracle-dsn) or (--oracle-wallet-path + --oracle-tns-alias), each flag settable by its ATP_* environment variable instead (omit user/password when --oracle-external-auth is set) |
Environment Variables¶
You can provide Oracle credentials via environment variables (both the Rust binary and the Python launcher read them):
ATP_WALLET_PATHATP_TNS_ALIASATP_DSNATP_USERATP_PASSWORDATP_EXTERNAL_AUTHATP_POOL_MINATP_POOL_MAXATP_POOL_TIMEOUT_SECS
POSTGRES_POOL_MAX, POSTGRES_OP_TIMEOUT_SECS, REDIS_POOL_MAX, REDIS_RETENTION_DAYS, and REDIS_OP_TIMEOUT_SECS are read only by the Python launcher, as defaults for its pool, timeout, and retention flags; the Rust binary takes those settings from flags alone.
Store URLs from the environment or a file¶
A PostgreSQL or Redis connection URL can carry the store's password, and a URL on the command line is readable by other processes (/proc/<pid>/cmdline, process listings, a Deployment's args). Both the Rust smg binary and the Python launcher also read each URL from an environment variable, or from a file such as a mounted Kubernetes Secret:
| Setting | Flag | Environment | File flag | File environment |
|---|---|---|---|---|
| PostgreSQL URL | --postgres-db-url |
SMG_POSTGRES_DB_URL |
--postgres-db-url-file |
SMG_POSTGRES_DB_URL_FILE |
| Redis URL | --redis-url |
SMG_REDIS_URL |
--redis-url-file |
SMG_REDIS_URL_FILE |
- The flag beats the environment. A URL (from the flag or the environment) together with a URL file for the same store is refused at startup rather than ranked.
- A URL file must hold the URL alone. Surrounding whitespace is dropped from the file's content and from a URL set in the environment, including the trailing newline a Secret made from a file carries. An empty file, or one holding more than one line, is refused.
- A URL file is read only for the selected
--history-backend: a--redis-url-filepassed to apostgresstart is ignored, not opened. - The Python launcher also accepts the older
POSTGRES_DB_URLandREDIS_URLnames when theSMG_names are unset; the Rust binary reads only theSMG_names. - No URL is echoed: a refusal names the sources and the file path, never the value. The Rust binary's startup log (
CREDENTIAL SOURCES) records whether each URL came from a flag, the environment, or a file; the Python launcher prints no such record.
Verify¶
curl http://localhost:30000/healthA missing or invalid required setting (for example the DB URL or Oracle credentials) fails startup with a config validation error.
Every remote backend is contacted at startup, and SMG refuses to start when its store cannot be reached. For PostgreSQL and Redis, a store that is unreachable, or that accepts connections but does not answer, stops startup with an error naming the backend and the store URL (password masked). PostgreSQL and Oracle also create tables and check migrations at startup, so pending migrations stop startup too (see Schema migrations).
Next Steps¶
- Chat History Concepts — backend architecture and tradeoffs
- Configuration Reference — full storage flag reference