Fionn TolsterandGitHub
d7549471ab
## Pull Request Fixes https://github.com/SigNoz/signoz-otel-collector/issues/750 ### Summary We have a usecase for ingesting historical logs in batches. We handle the logs generation, but it turns out the Clickhouselogsexporter drops logs that are 15 days old. This is hardcoded. This PR simply makes this value configurable. ```log 2026-01-08T21:16:22.145Z debug clickhouselogsexporter/exporter.go:565 skipping log {"resource": {"service.instance.id": "2b9dd48d-28c9-4077-acc9-d61f6750c381", "service.name": "/signoz-otel-collector", "service.version": "dev"}, "otelcol.component.id": "clickhouselogsexporter", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "ts": 1766154813095000000, "oldestAllowedTs": 1766610982140734761} ``` ### Resource Impact **Will this change affect the application's resource usage?** - [ ] Yes - [x] No
2.7 KiB
ClickHouse Logs Exporter
This exporter supports sending OpenTelemetry logs to ClickHouse.
ClickHouse is an open-source, high performance columnar OLAP database management system for real-time analytics using SQL. Throughput can be measured in rows per second or megabytes per second. If the data is placed in the page cache, a query that is not too complex is processed on modern hardware at a speed of approximately 2-10 GB/s of uncompressed data on a single server. If 10 bytes of columns are extracted, the speed is expected to be around 100-200 million rows per second.
Note: Always add batch-processor to collector pipeline, as ClickHouse document says:
We recommend inserting data in packets of at least 1000 rows, or no more than a single request per second. When inserting to a MergeTree table from a tab-separated dump, the insertion speed can be from 50 to 200 MB/s.
Configuration options
The following settings are required:
dsn(no default): The ClickHouse server DSN (Data Source Name), for exampletcp://127.0.0.1:9000?username=user&password=qwertyFor tcp protocol reference: ClickHouse/clickhouse-go#dsn. For http protocol reference: mailru/go-clickhouse/#dsn.
The following settings can be optionally configured:
timeout(default = 5s): The timeout for every attempt to send data to the backend.max_allowed_data_age_days(default = 15): Drop logs older than now minus this many days.sending_queuequeue_size(default = 5000): Maximum number of batches kept in memory before dropping data.
retry_on_failureenabled(default = true)initial_interval(default = 5s): The Time to wait after the first failure before retrying; ignored ifenabledisfalsemax_interval(default = 30s): The upper bound on backoff; ignored ifenabledisfalsemax_elapsed_time(default = 300s): The maximum amount of time spent trying to send a batch; ignored ifenabledisfalse
Example
receivers:
nop:
processors:
batch:
timeout: 10s
exporters:
clickhouselogsexporter:
dsn: tcp://127.0.0.1:9000
timeout: 5s
max_allowed_data_age_days: 15
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
service:
pipelines:
logs:
receivers: [nop]
processors: [batch]
exporters: [clickhouselogsexporter]