zeekay
bbc229b40e
Rename the two exporters Hanzo owns to drop the "clickhouse" brand: - exporter/clickhousetracesexporter -> exporter/datastoretracesexporter (package, component id clickhousetraces -> datastoretraces) - exporter/clickhouselogsexporter -> exporter/datastorelogsexporter (package, component id clickhouselogsexporter -> datastorelogsexporter) - symbols: WithClickHouseClient -> WithDatastoreClient, newClickhouseClient -> newDatastoreClient, pushToClickhouse -> pushToDatastore, clickhouseTracesExporter -> datastoreTracesExporter; files clickhouse_exporter*.go -> datastore_exporter*.go. Left intact (external, not ours): github.com/ClickHouse/clickhouse-go/v2 wire driver (clickhouse.Open/Conn/ParseDSN) and github.com/srikanthccv/ ClickHouse-go-mock test helper (cmock.NewClickHouseNative / NewClickHouseWithQueryMatcher). Sibling clickhousemetricswrite exporter and clickhousesystemtablesreceiver are out of scope. go build ./components/ and go vet of both packages pass.
2.7 KiB
ClickHouse Logs Exporter
This exporter supports sending OpenTelemetry logs to ClickHouse.
ClickHouse is an open-source, high performance columnar OLAP database management system for real-time analytics using SQL. Throughput can be measured in rows per second or megabytes per second. If the data is placed in the page cache, a query that is not too complex is processed on modern hardware at a speed of approximately 2-10 GB/s of uncompressed data on a single server. If 10 bytes of columns are extracted, the speed is expected to be around 100-200 million rows per second.
Note: Always add batch-processor to collector pipeline, as ClickHouse document says:
We recommend inserting data in packets of at least 1000 rows, or no more than a single request per second. When inserting to a MergeTree table from a tab-separated dump, the insertion speed can be from 50 to 200 MB/s.
Configuration options
The following settings are required:
dsn(no default): The ClickHouse server DSN (Data Source Name), for exampletcp://127.0.0.1:9000?username=user&password=qwertyFor tcp protocol reference: ClickHouse/clickhouse-go#dsn. For http protocol reference: mailru/go-clickhouse/#dsn.
The following settings can be optionally configured:
timeout(default = 5s): The timeout for every attempt to send data to the backend.max_allowed_data_age_days(default = 15): Drop logs older than now minus this many days.sending_queuequeue_size(default = 5000): Maximum number of batches kept in memory before dropping data.
retry_on_failureenabled(default = true)initial_interval(default = 5s): The Time to wait after the first failure before retrying; ignored ifenabledisfalsemax_interval(default = 30s): The upper bound on backoff; ignored ifenabledisfalsemax_elapsed_time(default = 300s): The maximum amount of time spent trying to send a batch; ignored ifenabledisfalse
Example
receivers:
nop:
processors:
batch:
timeout: 10s
exporters:
datastorelogsexporter:
dsn: tcp://127.0.0.1:9000
timeout: 5s
max_allowed_data_age_days: 15
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
service:
pipelines:
logs:
receivers: [nop]
processors: [batch]
exporters: [datastorelogsexporter]