zooqueen
c6d775009c
metadata.yaml declared clickhousesystemtablesreceiver and o11yclickhousemeter while the factories register datastoresystemtablesreceiver and o11ydatastoremeter. The receiver's testdata was keyed on the old name, so TestLoadConfig could not find its config and failed on empty dsn. o11ydatastoremeter's second expectation still read go.o11y.io where the exporter emits go.o11y.hanzo.ai; the first had been updated. READMEs, the migrator DSN and the testdata hosts follow. NOTICE keeps the upstream attribution.
2.7 KiB
Datastore Logs Exporter
This exporter supports sending OpenTelemetry logs to ClickHouse.
Datastore is an open-source, high performance columnar OLAP database management system for real-time analytics using SQL. Throughput can be measured in rows per second or megabytes per second. If the data is placed in the page cache, a query that is not too complex is processed on modern hardware at a speed of approximately 2-10 GB/s of uncompressed data on a single server. If 10 bytes of columns are extracted, the speed is expected to be around 100-200 million rows per second.
Note: Always add batch-processor to collector pipeline, as ClickHouse document says:
We recommend inserting data in packets of at least 1000 rows, or no more than a single request per second. When inserting to a MergeTree table from a tab-separated dump, the insertion speed can be from 50 to 200 MB/s.
Configuration options
The following settings are required:
dsn(no default): The Datastore server DSN (Data Source Name), for exampletcp://127.0.0.1:9000?username=user&password=qwertyFor tcp protocol reference: Datastore/clickhouse-go#dsn. For http protocol reference: mailru/go-clickhouse/#dsn.
The following settings can be optionally configured:
timeout(default = 5s): The timeout for every attempt to send data to the backend.max_allowed_data_age_days(default = 15): Drop logs older than now minus this many days.sending_queuequeue_size(default = 5000): Maximum number of batches kept in memory before dropping data.
retry_on_failureenabled(default = true)initial_interval(default = 5s): The Time to wait after the first failure before retrying; ignored ifenabledisfalsemax_interval(default = 30s): The upper bound on backoff; ignored ifenabledisfalsemax_elapsed_time(default = 300s): The maximum amount of time spent trying to send a batch; ignored ifenabledisfalse
Example
receivers:
nop:
processors:
batch:
timeout: 10s
exporters:
datastorelogsexporter:
dsn: tcp://127.0.0.1:9000
timeout: 5s
max_allowed_data_age_days: 15
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
service:
pipelines:
logs:
receivers: [nop]
processors: [batch]
exporters: [datastorelogsexporter]