From 5f7be1f66679d87724f17225fb5c457ff06b2c1c Mon Sep 17 00:00:00 2001 From: mikeua <2239742+mikeua@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:57:14 -0400 Subject: [PATCH] docs: document log store configuration via [log_store] in config.toml The log store guide and the environment variable reference described --log-store-* flags and GRAPH_LOG_STORE_* environment variables that do not exist; the log store is only configurable through the [log_store] section of the TOML config (#6278). Replace those examples with TOML, document every [log_store] key with its defaults, drop the nonexistent max-size/retention-days options in favour of retention_hours, and note that _logs returns a NotSupported error when no store is configured (#6637). Also update the matching startup message. --- docs/environment-variables.md | 39 +----- docs/log-store.md | 222 ++++++++++------------------------ node/src/launcher.rs | 2 +- 3 files changed, 68 insertions(+), 195 deletions(-) diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 7a601e32e64..36ac6176747 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -322,43 +322,14 @@ those. **For complete log store documentation**, including detailed configuration, querying examples, and choosing the right backend, see the **[Log Store Guide](log-store.md)**. -### Quick Reference - -**Backend selection:** -- `GRAPH_LOG_STORE_BACKEND`: `disabled` (default), `elasticsearch`, `loki`, or `file` - -**Elasticsearch:** -- `GRAPH_LOG_STORE_ELASTICSEARCH_URL`: Elasticsearch endpoint URL (required) -- `GRAPH_LOG_STORE_ELASTICSEARCH_USER`: Username (optional) -- `GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD`: Password (optional) -- `GRAPH_LOG_STORE_ELASTICSEARCH_INDEX`: Index name (default: `subgraph`) - -**Loki:** -- `GRAPH_LOG_STORE_LOKI_URL`: Loki endpoint URL (required) -- `GRAPH_LOG_STORE_LOKI_TENANT_ID`: Tenant ID (optional) - -**File-based:** -- `GRAPH_LOG_STORE_FILE_DIR`: Log directory (required) -- `GRAPH_LOG_STORE_FILE_MAX_SIZE`: Max file size in bytes (default: 104857600 = 100MB) -- `GRAPH_LOG_STORE_FILE_RETENTION_DAYS`: Retention period (default: 30) - -**Deprecated variables** (will be removed in future versions): -- `GRAPH_ELASTICSEARCH_URL` → use `GRAPH_LOG_STORE_ELASTICSEARCH_URL` -- `GRAPH_ELASTICSEARCH_USER` → use `GRAPH_LOG_STORE_ELASTICSEARCH_USER` -- `GRAPH_ELASTICSEARCH_PASSWORD` → use `GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD` -- `GRAPH_ELASTIC_SEARCH_INDEX` → use `GRAPH_LOG_STORE_ELASTICSEARCH_INDEX` +The log store is configured in the `[log_store]` section of the TOML configuration file passed with `--config`, not through environment variables. The former `GRAPH_ELASTICSEARCH_*` / `ELASTICSEARCH_*` variables and `--elasticsearch-*` flags have been removed. ### Example: File-based Logs for Local Development -```bash -mkdir -p ./graph-logs -export GRAPH_LOG_STORE_BACKEND=file -export GRAPH_LOG_STORE_FILE_DIR=./graph-logs - -graph-node \ - --postgres-url postgresql://graph:pass@localhost/graph-node \ - --ethereum-rpc mainnet:https://... \ - --ipfs 127.0.0.1:5001 +```toml +[log_store] +backend = "file" +directory = "./graph-logs" ``` See the **[Log Store Guide](log-store.md)** for: diff --git a/docs/log-store.md b/docs/log-store.md index 8be1cddecd8..b3b39a54060 100644 --- a/docs/log-store.md +++ b/docs/log-store.md @@ -12,11 +12,9 @@ This guide explains how to configure subgraph indexing logs storage in graph-nod - [Loki](#loki) - [Disabled](#disabled) - [Configuration](#configuration) - - [Environment Variables](#environment-variables) - - [CLI Arguments](#cli-arguments) - - [Configuration Precedence](#configuration-precedence) + - [The `[log_store]` section](#the-log_store-section) + - [Migrating from earlier configuration](#migrating-from-earlier-configuration) - [Querying Logs](#querying-logs) -- [Migrating from Deprecated Configuration](#migrating-from-deprecated-configuration) - [Choosing the Right Backend](#choosing-the-right-backend) - [Best Practices](#best-practices) - [Troubleshooting](#troubleshooting) @@ -130,22 +128,13 @@ File-based logs stream through files line-by-line with bounded memory usage. #### Configuration -**Minimum configuration (CLI):** -```bash -graph-node \ - --postgres-url postgresql://graph:pass@localhost/graph-node \ - --ethereum-rpc mainnet:https://... \ - --ipfs 127.0.0.1:5001 \ - --log-store-backend file \ - --log-store-file-dir ./graph-logs -``` +Add a `[log_store]` section to the `graph-node.toml` file passed with `--config`: -**Full configuration (environment variables):** -```bash -export GRAPH_LOG_STORE_BACKEND=file -export GRAPH_LOG_STORE_FILE_DIR=/var/log/graph-node -export GRAPH_LOG_STORE_FILE_MAX_SIZE=104857600 # 100MB -export GRAPH_LOG_STORE_FILE_RETENTION_DAYS=30 +```toml +[log_store] +backend = "file" +directory = "./graph-logs" +retention_hours = 72 # optional; 0 (the default) keeps logs forever ``` #### Features @@ -160,7 +149,7 @@ export GRAPH_LOG_STORE_FILE_RETENTION_DAYS=30 **Limitations:** - Not suitable for production with high log volume - No indexing (O(n) query time scales with file size) -- No automatic log rotation or retention management +- No log rotation; the only cleanup is the optional `retention_hours` - Single file per subgraph (no sharding) #### When to Use @@ -202,36 +191,22 @@ graph-node → Elasticsearch HTTP API → Elasticsearch cluster #### Configuration -**Minimum configuration (CLI):** -```bash -graph-node \ - --postgres-url postgresql://graph:pass@localhost/graph-node \ - --ethereum-rpc mainnet:https://... \ - --ipfs 127.0.0.1:5001 \ - --log-store-backend elasticsearch \ - --log-store-elasticsearch-url http://localhost:9200 +**Minimum configuration:** +```toml +[log_store] +backend = "elasticsearch" +url = "http://localhost:9200" ``` **Full configuration with authentication:** -```bash -graph-node \ - --postgres-url postgresql://graph:pass@localhost/graph-node \ - --ethereum-rpc mainnet:https://... \ - --ipfs 127.0.0.1:5001 \ - --log-store-backend elasticsearch \ - --log-store-elasticsearch-url https://es.example.com:9200 \ - --log-store-elasticsearch-user elastic \ - --log-store-elasticsearch-password secret \ - --log-store-elasticsearch-index subgraph-logs -``` - -**Environment variables:** -```bash -export GRAPH_LOG_STORE_BACKEND=elasticsearch -export GRAPH_LOG_STORE_ELASTICSEARCH_URL=http://localhost:9200 -export GRAPH_LOG_STORE_ELASTICSEARCH_USER=elastic -export GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD=secret -export GRAPH_LOG_STORE_ELASTICSEARCH_INDEX=subgraph-logs +```toml +[log_store] +backend = "elasticsearch" +url = "https://es.example.com:9200" +username = "elastic" +password = "secret" +index = "subgraph-logs" # default: "subgraph" +timeout_secs = 10 # default: 10 ``` #### Index Configuration @@ -300,32 +275,21 @@ graph-node → Loki HTTP API → Loki #### Configuration -**Minimum configuration (CLI):** -```bash -graph-node \ - --postgres-url postgresql://graph:pass@localhost/graph-node \ - --ethereum-rpc mainnet:https://... \ - --ipfs 127.0.0.1:5001 \ - --log-store-backend loki \ - --log-store-loki-url http://localhost:3100 +**Minimum configuration:** +```toml +[log_store] +backend = "loki" +url = "http://localhost:3100" ``` -**With multi-tenancy:** -```bash -graph-node \ - --postgres-url postgresql://graph:pass@localhost/graph-node \ - --ethereum-rpc mainnet:https://... \ - --ipfs 127.0.0.1:5001 \ - --log-store-backend loki \ - --log-store-loki-url http://localhost:3100 \ - --log-store-loki-tenant-id my-graph-node -``` - -**Environment variables:** -```bash -export GRAPH_LOG_STORE_BACKEND=loki -export GRAPH_LOG_STORE_LOKI_URL=http://localhost:3100 -export GRAPH_LOG_STORE_LOKI_TENANT_ID=my-graph-node +**With multi-tenancy and authentication:** +```toml +[log_store] +backend = "loki" +url = "http://localhost:3100" +tenant_id = "my-graph-node" +username = "loki" # optional +password = "secret" # optional ``` #### Labels @@ -364,25 +328,19 @@ When log storage is disabled (the default), subgraph logs are **still written to **Important:** "Disabled" does NOT mean logs are discarded. It means: - Logs appear in stdout/stderr (traditional behavior) - Logs are not stored in a separate queryable backend -- The `_logs` GraphQL query returns empty results +- The `_logs` GraphQL query returns a "not supported" error This is the default behavior - logs continue to work exactly as they did before this feature was added. #### Configuration **Explicitly disable:** -```bash -export GRAPH_LOG_STORE_BACKEND=disabled +```toml +[log_store] +backend = "disabled" ``` -**Or simply don't configure a backend** (defaults to disabled): -```bash -# No log store configuration = disabled -graph-node \ - --postgres-url postgresql://graph:pass@localhost/graph-node \ - --ethereum-rpc mainnet:https://... \ - --ipfs 127.0.0.1:5001 -``` +**Or simply omit the `[log_store]` section** (defaults to disabled). #### Features @@ -393,7 +351,7 @@ graph-node \ - Logs still appear in stdout/stderr for debugging **Limitations:** -- Cannot query logs via GraphQL (`_logs` returns empty results) +- Cannot query logs via GraphQL (`_logs` returns a "not supported" error) - No separation of subgraph logs from other graph-node logs in stdout - Logs mixed with system logs (harder to filter programmatically) - No structured querying or filtering capabilities @@ -407,82 +365,27 @@ Use disabled log storage when: ## Configuration -### Environment Variables - -Environment variables are the recommended way to configure log stores, especially in containerized deployments. - -#### Backend Selection - -```bash -GRAPH_LOG_STORE_BACKEND= -``` -Valid values: `disabled`, `elasticsearch`, `loki`, `file` +### The `[log_store]` section -#### Elasticsearch - -```bash -GRAPH_LOG_STORE_ELASTICSEARCH_URL=http://localhost:9200 -GRAPH_LOG_STORE_ELASTICSEARCH_USER=elastic # Optional -GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD=secret # Optional -GRAPH_LOG_STORE_ELASTICSEARCH_INDEX=subgraph # Default: "subgraph" -``` - -#### Loki +The log store is configured in the `[log_store]` section of the TOML configuration file passed to `graph-node` with `--config` (see [config.md](config.md)). There are no command-line flags or environment variables for it. -```bash -GRAPH_LOG_STORE_LOKI_URL=http://localhost:3100 -GRAPH_LOG_STORE_LOKI_TENANT_ID=my-tenant # Optional -``` - -#### File - -```bash -GRAPH_LOG_STORE_FILE_DIR=/var/log/graph-node -GRAPH_LOG_STORE_FILE_MAX_SIZE=104857600 # Default: 100MB -GRAPH_LOG_STORE_FILE_RETENTION_DAYS=30 # Default: 30 -``` - -### CLI Arguments - -CLI arguments provide the same functionality as environment variables and the two can be mixed together. - -#### Backend Selection - -```bash ---log-store-backend -``` - -#### Elasticsearch - -```bash ---log-store-elasticsearch-url ---log-store-elasticsearch-user ---log-store-elasticsearch-password ---log-store-elasticsearch-index -``` - -#### Loki - -```bash ---log-store-loki-url ---log-store-loki-tenant-id -``` - -#### File - -```bash ---log-store-file-dir ---log-store-file-max-size ---log-store-file-retention-days -``` +| Key | Backends | Required | Default | Description | +|-----|----------|----------|---------|-------------| +| `backend` | all | yes | | `disabled` (or `none`), `elasticsearch` (or `elastic`, `es`), `loki`, `file` (or `files`) | +| `url` | elasticsearch, loki | yes | | Endpoint URL | +| `username` | elasticsearch, loki | no | | Basic-auth user name | +| `password` | elasticsearch, loki | no | | Basic-auth password | +| `index` | elasticsearch | no | `subgraph` | Index name | +| `timeout_secs` | elasticsearch | no | `10` | Request timeout in seconds | +| `tenant_id` | loki | no | | Loki tenant ID (`X-Scope-OrgID`) | +| `directory` | file | yes | | Directory for the `.jsonl` files | +| `retention_hours` | file | no | `0` | Delete entries older than this many hours; `0` keeps them forever | -### Configuration Precedence +If the section is missing, the log store is disabled. An invalid section (for example, a missing `url`) is logged as a warning at startup, and the log store is disabled. -When multiple configuration methods are used: +### Migrating from earlier configuration -1. **CLI arguments** take highest precedence -2. **Environment variables** are used if no CLI args provided -3. **Defaults** are used if neither is set +The `--elasticsearch-url`, `--elasticsearch-user` and `--elasticsearch-password` flags and the matching `ELASTICSEARCH_*` / `GRAPH_ELASTICSEARCH_*` environment variables have been removed. Move those settings into a `[log_store]` section with `backend = "elasticsearch"`, as shown above. ## Querying Logs @@ -792,23 +695,23 @@ cat graph-logs/QmExample.jsonl | jq 'select(.text | contains("timeout"))' ### File-based Logs **Problem: Log file doesn't exist** -- Check `GRAPH_LOG_STORE_FILE_DIR` is set correctly +- Check `directory` in the `[log_store]` section is set correctly - Verify directory is writable by graph-node **Problem: Queries are slow** - Subgraph logs file may be very large -- Consider archiving old logs or implementing retention +- Consider archiving old logs or setting `retention_hours` - For high-volume production use, switch to Elasticsearch or Loki **Problem: Disk filling up** -- Implement log rotation +- Set `retention_hours` in the `[log_store]` section - Reduce log verbosity in subgraph code - Set up monitoring for disk usage ### Elasticsearch **Problem: Cannot connect to Elasticsearch** -- Verify `GRAPH_LOG_STORE_ELASTICSEARCH_URL` is correct +- Verify `url` in the `[log_store]` section is correct - Check Elasticsearch is running: `curl http://localhost:9200` - Verify authentication credentials if using security features - Check network connectivity and firewall rules @@ -828,7 +731,7 @@ cat graph-logs/QmExample.jsonl | jq 'select(.text | contains("timeout"))' ### Loki **Problem: Cannot connect to Loki** -- Verify `GRAPH_LOG_STORE_LOKI_URL` is correct +- Verify `url` in the `[log_store]` section is correct - Check Loki is running: `curl http://localhost:3100/ready` - Verify tenant ID if using multi-tenancy - Check network connectivity @@ -847,7 +750,6 @@ cat graph-logs/QmExample.jsonl | jq 'select(.text | contains("timeout"))' ## Further Reading -- [Environment Variables Reference](environment-variables.md) - [Graph Node Configuration](config.md) - [Elasticsearch Documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html) - [Grafana Loki Documentation](https://grafana.com/docs/loki/latest/) diff --git a/node/src/launcher.rs b/node/src/launcher.rs index 18ee4c727db..e693ee28e8a 100644 --- a/node/src/launcher.rs +++ b/node/src/launcher.rs @@ -478,7 +478,7 @@ pub async fn run( None => { info!( logger, - "No [log_store] section in config, log queries will return empty results" + "No [log_store] section in config, log queries are not supported" ); ( Arc::new(graph::components::log_store::NoOpLogStore)