Skip to content

docs: document log store configuration via [log_store] #6739

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mikeua wants to merge 1 commit into
base: master
Choose a base branch
from
+68 −195
Open

docs: document log store configuration via [log_store] #6739

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 5 additions & 34 deletions docs/environment-variables.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,43 +322,14 @@ those.

**For complete log store documentation**, including detailed configuration, querying examples, and choosing the right backend, see the **[Log Store Guide](log-store.md)**.

### Quick Reference

**Backend selection:**
- `GRAPH_LOG_STORE_BACKEND`: `disabled` (default), `elasticsearch`, `loki`, or `file`

**Elasticsearch:**
- `GRAPH_LOG_STORE_ELASTICSEARCH_URL`: Elasticsearch endpoint URL (required)
- `GRAPH_LOG_STORE_ELASTICSEARCH_USER`: Username (optional)
- `GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD`: Password (optional)
- `GRAPH_LOG_STORE_ELASTICSEARCH_INDEX`: Index name (default: `subgraph`)

**Loki:**
- `GRAPH_LOG_STORE_LOKI_URL`: Loki endpoint URL (required)
- `GRAPH_LOG_STORE_LOKI_TENANT_ID`: Tenant ID (optional)

**File-based:**
- `GRAPH_LOG_STORE_FILE_DIR`: Log directory (required)
- `GRAPH_LOG_STORE_FILE_MAX_SIZE`: Max file size in bytes (default: 104857600 = 100MB)
- `GRAPH_LOG_STORE_FILE_RETENTION_DAYS`: Retention period (default: 30)

**Deprecated variables** (will be removed in future versions):
- `GRAPH_ELASTICSEARCH_URL` → use `GRAPH_LOG_STORE_ELASTICSEARCH_URL`
- `GRAPH_ELASTICSEARCH_USER` → use `GRAPH_LOG_STORE_ELASTICSEARCH_USER`
- `GRAPH_ELASTICSEARCH_PASSWORD` → use `GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD`
- `GRAPH_ELASTIC_SEARCH_INDEX` → use `GRAPH_LOG_STORE_ELASTICSEARCH_INDEX`
The log store is configured in the `[log_store]` section of the TOML configuration file passed with `--config`, not through environment variables. The former `GRAPH_ELASTICSEARCH_*` / `ELASTICSEARCH_*` variables and `--elasticsearch-*` flags have been removed.

### Example: File-based Logs for Local Development

```bash
mkdir -p ./graph-logs
export GRAPH_LOG_STORE_BACKEND=file
export GRAPH_LOG_STORE_FILE_DIR=./graph-logs

graph-node \
--postgres-url postgresql://graph:pass@localhost/graph-node \
--ethereum-rpc mainnet:https://... \
--ipfs 127.0.0.1:5001
```toml
[log_store]
backend = "file"
directory = "./graph-logs"
```

See the **[Log Store Guide](log-store.md)** for:
Expand Down
222 changes: 62 additions & 160 deletions docs/log-store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,9 @@ This guide explains how to configure subgraph indexing logs storage in graph-nod
- [Loki](#loki)
- [Disabled](#disabled)
- [Configuration](#configuration)
- [Environment Variables](#environment-variables)
- [CLI Arguments](#cli-arguments)
- [Configuration Precedence](#configuration-precedence)
- [The `[log_store]` section](#the-log_store-section)
- [Migrating from earlier configuration](#migrating-from-earlier-configuration)
- [Querying Logs](#querying-logs)
- [Migrating from Deprecated Configuration](#migrating-from-deprecated-configuration)
- [Choosing the Right Backend](#choosing-the-right-backend)
- [Best Practices](#best-practices)
- [Troubleshooting](#troubleshooting)
Expand Down Expand Up @@ -130,22 +128,13 @@ File-based logs stream through files line-by-line with bounded memory usage.

#### Configuration

**Minimum configuration (CLI):**
```bash
graph-node \
--postgres-url postgresql://graph:pass@localhost/graph-node \
--ethereum-rpc mainnet:https://... \
--ipfs 127.0.0.1:5001 \
--log-store-backend file \
--log-store-file-dir ./graph-logs
```
Add a `[log_store]` section to the `graph-node.toml` file passed with `--config`:

**Full configuration (environment variables):**
```bash
export GRAPH_LOG_STORE_BACKEND=file
export GRAPH_LOG_STORE_FILE_DIR=/var/log/graph-node
export GRAPH_LOG_STORE_FILE_MAX_SIZE=104857600 # 100MB
export GRAPH_LOG_STORE_FILE_RETENTION_DAYS=30
```toml
[log_store]
backend = "file"
directory = "./graph-logs"
retention_hours = 72 # optional; 0 (the default) keeps logs forever
```

#### Features
Expand All @@ -160,7 +149,7 @@ export GRAPH_LOG_STORE_FILE_RETENTION_DAYS=30
**Limitations:**
- Not suitable for production with high log volume
- No indexing (O(n) query time scales with file size)
- No automatic log rotation or retention management
- No log rotation; the only cleanup is the optional `retention_hours`
- Single file per subgraph (no sharding)

#### When to Use
Expand Down Expand Up @@ -202,36 +191,22 @@ graph-node → Elasticsearch HTTP API → Elasticsearch cluster

#### Configuration

**Minimum configuration (CLI):**
```bash
graph-node \
--postgres-url postgresql://graph:pass@localhost/graph-node \
--ethereum-rpc mainnet:https://... \
--ipfs 127.0.0.1:5001 \
--log-store-backend elasticsearch \
--log-store-elasticsearch-url http://localhost:9200
**Minimum configuration:**
```toml
[log_store]
backend = "elasticsearch"
url = "http://localhost:9200"
```

**Full configuration with authentication:**
```bash
graph-node \
--postgres-url postgresql://graph:pass@localhost/graph-node \
--ethereum-rpc mainnet:https://... \
--ipfs 127.0.0.1:5001 \
--log-store-backend elasticsearch \
--log-store-elasticsearch-url https://es.example.com:9200 \
--log-store-elasticsearch-user elastic \
--log-store-elasticsearch-password secret \
--log-store-elasticsearch-index subgraph-logs
```

**Environment variables:**
```bash
export GRAPH_LOG_STORE_BACKEND=elasticsearch
export GRAPH_LOG_STORE_ELASTICSEARCH_URL=http://localhost:9200
export GRAPH_LOG_STORE_ELASTICSEARCH_USER=elastic
export GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD=secret
export GRAPH_LOG_STORE_ELASTICSEARCH_INDEX=subgraph-logs
```toml
[log_store]
backend = "elasticsearch"
url = "https://es.example.com:9200"
username = "elastic"
password = "secret"
index = "subgraph-logs" # default: "subgraph"
timeout_secs = 10 # default: 10
```

#### Index Configuration
Expand Down Expand Up @@ -300,32 +275,21 @@ graph-node → Loki HTTP API → Loki

#### Configuration

**Minimum configuration (CLI):**
```bash
graph-node \
--postgres-url postgresql://graph:pass@localhost/graph-node \
--ethereum-rpc mainnet:https://... \
--ipfs 127.0.0.1:5001 \
--log-store-backend loki \
--log-store-loki-url http://localhost:3100
**Minimum configuration:**
```toml
[log_store]
backend = "loki"
url = "http://localhost:3100"
```

**With multi-tenancy:**
```bash
graph-node \
--postgres-url postgresql://graph:pass@localhost/graph-node \
--ethereum-rpc mainnet:https://... \
--ipfs 127.0.0.1:5001 \
--log-store-backend loki \
--log-store-loki-url http://localhost:3100 \
--log-store-loki-tenant-id my-graph-node
```

**Environment variables:**
```bash
export GRAPH_LOG_STORE_BACKEND=loki
export GRAPH_LOG_STORE_LOKI_URL=http://localhost:3100
export GRAPH_LOG_STORE_LOKI_TENANT_ID=my-graph-node
**With multi-tenancy and authentication:**
```toml
[log_store]
backend = "loki"
url = "http://localhost:3100"
tenant_id = "my-graph-node"
username = "loki" # optional
password = "secret" # optional
```

#### Labels
Expand Down Expand Up @@ -364,25 +328,19 @@ When log storage is disabled (the default), subgraph logs are **still written to
**Important:** "Disabled" does NOT mean logs are discarded. It means:
- Logs appear in stdout/stderr (traditional behavior)
- Logs are not stored in a separate queryable backend
- The `_logs` GraphQL query returns empty results
- The `_logs` GraphQL query returns a "not supported" error

This is the default behavior - logs continue to work exactly as they did before this feature was added.

#### Configuration

**Explicitly disable:**
```bash
export GRAPH_LOG_STORE_BACKEND=disabled
```toml
[log_store]
backend = "disabled"
```

**Or simply don't configure a backend** (defaults to disabled):
```bash
# No log store configuration = disabled
graph-node \
--postgres-url postgresql://graph:pass@localhost/graph-node \
--ethereum-rpc mainnet:https://... \
--ipfs 127.0.0.1:5001
```
**Or simply omit the `[log_store]` section** (defaults to disabled).

#### Features

Expand All @@ -393,7 +351,7 @@ graph-node \
- Logs still appear in stdout/stderr for debugging

**Limitations:**
- Cannot query logs via GraphQL (`_logs` returns empty results)
- Cannot query logs via GraphQL (`_logs` returns a "not supported" error)
- No separation of subgraph logs from other graph-node logs in stdout
- Logs mixed with system logs (harder to filter programmatically)
- No structured querying or filtering capabilities
Expand All @@ -407,82 +365,27 @@ Use disabled log storage when:

## Configuration

### Environment Variables

Environment variables are the recommended way to configure log stores, especially in containerized deployments.

#### Backend Selection

```bash
GRAPH_LOG_STORE_BACKEND=<backend>
```
Valid values: `disabled`, `elasticsearch`, `loki`, `file`
### The `[log_store]` section

#### Elasticsearch

```bash
GRAPH_LOG_STORE_ELASTICSEARCH_URL=http://localhost:9200
GRAPH_LOG_STORE_ELASTICSEARCH_USER=elastic # Optional
GRAPH_LOG_STORE_ELASTICSEARCH_PASSWORD=secret # Optional
GRAPH_LOG_STORE_ELASTICSEARCH_INDEX=subgraph # Default: "subgraph"
```

#### Loki
The log store is configured in the `[log_store]` section of the TOML configuration file passed to `graph-node` with `--config` (see [config.md](config.md)). There are no command-line flags or environment variables for it.

```bash
GRAPH_LOG_STORE_LOKI_URL=http://localhost:3100
GRAPH_LOG_STORE_LOKI_TENANT_ID=my-tenant # Optional
```

#### File

```bash
GRAPH_LOG_STORE_FILE_DIR=/var/log/graph-node
GRAPH_LOG_STORE_FILE_MAX_SIZE=104857600 # Default: 100MB
GRAPH_LOG_STORE_FILE_RETENTION_DAYS=30 # Default: 30
```

### CLI Arguments

CLI arguments provide the same functionality as environment variables and the two can be mixed together.

#### Backend Selection

```bash
--log-store-backend <backend>
```

#### Elasticsearch

```bash
--log-store-elasticsearch-url <URL>
--log-store-elasticsearch-user <USER>
--log-store-elasticsearch-password <PASSWORD>
--log-store-elasticsearch-index <INDEX>
```

#### Loki

```bash
--log-store-loki-url <URL>
--log-store-loki-tenant-id <TENANT_ID>
```

#### File

```bash
--log-store-file-dir <DIR>
--log-store-file-max-size <BYTES>
--log-store-file-retention-days <DAYS>
```
| Key | Backends | Required | Default | Description |
|-----|----------|----------|---------|-------------|
| `backend` | all | yes | | `disabled` (or `none`), `elasticsearch` (or `elastic`, `es`), `loki`, `file` (or `files`) |
| `url` | elasticsearch, loki | yes | | Endpoint URL |
| `username` | elasticsearch, loki | no | | Basic-auth user name |
| `password` | elasticsearch, loki | no | | Basic-auth password |
| `index` | elasticsearch | no | `subgraph` | Index name |
| `timeout_secs` | elasticsearch | no | `10` | Request timeout in seconds |
| `tenant_id` | loki | no | | Loki tenant ID (`X-Scope-OrgID`) |
| `directory` | file | yes | | Directory for the `.jsonl` files |
| `retention_hours` | file | no | `0` | Delete entries older than this many hours; `0` keeps them forever |

### Configuration Precedence
If the section is missing, the log store is disabled. An invalid section (for example, a missing `url`) is logged as a warning at startup, and the log store is disabled.

When multiple configuration methods are used:
### Migrating from earlier configuration

1. **CLI arguments** take highest precedence
2. **Environment variables** are used if no CLI args provided
3. **Defaults** are used if neither is set
The `--elasticsearch-url`, `--elasticsearch-user` and `--elasticsearch-password` flags and the matching `ELASTICSEARCH_*` / `GRAPH_ELASTICSEARCH_*` environment variables have been removed. Move those settings into a `[log_store]` section with `backend = "elasticsearch"`, as shown above.

## Querying Logs

Expand Down Expand Up @@ -792,23 +695,23 @@ cat graph-logs/QmExample.jsonl | jq 'select(.text | contains("timeout"))'
### File-based Logs

**Problem: Log file doesn't exist**
- Check `GRAPH_LOG_STORE_FILE_DIR` is set correctly
- Check `directory` in the `[log_store]` section is set correctly
- Verify directory is writable by graph-node

**Problem: Queries are slow**
- Subgraph logs file may be very large
- Consider archiving old logs or implementing retention
- Consider archiving old logs or setting `retention_hours`
- For high-volume production use, switch to Elasticsearch or Loki

**Problem: Disk filling up**
- Implement log rotation
- Set `retention_hours` in the `[log_store]` section
- Reduce log verbosity in subgraph code
- Set up monitoring for disk usage

### Elasticsearch

**Problem: Cannot connect to Elasticsearch**
- Verify `GRAPH_LOG_STORE_ELASTICSEARCH_URL` is correct
- Verify `url` in the `[log_store]` section is correct
- Check Elasticsearch is running: `curl http://localhost:9200`
- Verify authentication credentials if using security features
- Check network connectivity and firewall rules
Expand All @@ -828,7 +731,7 @@ cat graph-logs/QmExample.jsonl | jq 'select(.text | contains("timeout"))'
### Loki

**Problem: Cannot connect to Loki**
- Verify `GRAPH_LOG_STORE_LOKI_URL` is correct
- Verify `url` in the `[log_store]` section is correct
- Check Loki is running: `curl http://localhost:3100/ready`
- Verify tenant ID if using multi-tenancy
- Check network connectivity
Expand All @@ -847,7 +750,6 @@ cat graph-logs/QmExample.jsonl | jq 'select(.text | contains("timeout"))'

## Further Reading

- [Environment Variables Reference](environment-variables.md)
- [Graph Node Configuration](config.md)
- [Elasticsearch Documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)
- [Grafana Loki Documentation](https://grafana.com/docs/loki/latest/)
2 changes: 1 addition & 1 deletion node/src/launcher.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,7 @@ pub async fn run(
None => {
info!(
logger,
"No [log_store] section in config, log queries will return empty results"
"No [log_store] section in config, log queries are not supported"
);
(
Arc::new(graph::components::log_store::NoOpLogStore)
Expand Down