mirror of
https://github.com/hanzoai/traces.git
synced 2026-11-03 04:05:45 +00:00
d5ff24815c
* [vParquet3] create new block encoding by copying vParquet2 * vParquet3: add dedicated columns to parquet schema and block meta (#2517) * Re-order schema to keep columns affected by column index changes low * Add spare columns for dedicated attributes to schema struct * Add dedicated column config to block meta * Read and write attributes in dedicated columns * Make order of dedicated attributes predictable when reading * Fix existing tests and benchmark * Run exiting benchmarks and tests with dedicated columns Co-authored-by: Mario <[email protected]> * Add dedicated columns to overrides module (#2551) * [vParquet3] Write path (#2555) * Add dedicated columns to overrides and blocks * Improvements * Change test * Fix tests * Extend ingester_test: * Add dedicated columns config to storage block * Review comments * Add comment * [vParquet3] dedicated columns read path (#2592) * Refactor and rename function blockMetaToDedicatedColumnMapping * Query dedicated attribute columns with TraceQL * Search tag values in dedicated attribute columns * Search tags in dedicated attribute columns * Search for values in dedicated attribute columns in tests * More consistent naming * Update block and meta.json in vparquet2/test-data * Test dedicated column in traceToParquet test * Format Go code * Introduce types for dedicated column type and scope Replace StaticTypeFromString() with DedicatedColumnType.ToStaticType() * The function dedicatedColumnsToColumnMapping() can receive multiple scopes * [vParquet3] Add support for dedicated columns in compactor (#2561) * Re-order schema to keep columns affected by column index changes low * Add spare columns for dedicated attributes to schema struct * Add dedicated column config to block meta * Read and write attributes in dedicated columns * Make order of dedicated attributes predictable when reading * Fix existing tests and benchmark * Run exiting benchmarks and tests with dedicated columns * Add dedicated columns to overrides and blocks * Support dedicated columns in compactor block selection * Changes to hash * More tests --------- Co-authored-by: A. Stoewer <[email protected]> * [vParquet3] pass dedicated columns to querier (#2603) * Add dedicated columns to SearchBlockRequest message * Assign SearchBlockRequest dedicated cols from BlockMeta and vice versa * Encode SearchBlockRequest to http request and vice versa * Don't add empty dedicated columns when building a search request * Unit tests with dedicated columns * Implement dedicated column scope and type as protobuf enums * [vParquet3] validate dedicated columns configuration (#2616) * Add validate function * Refactor: use DedicatedColumns type instead of []DedicatedColumn * Initialize logger before verifying the config This fixes the config verification output * Check for invalid dedicated columns with '-config.verify true' * Use ToTempopb() to validate dedicated column scope and type * [vParquet3] mention feature in CHANGELOG.md * [vParquet3] Address review comments * Remove TODO comment about caching the dedicated column hash * Shorten url param for dedicated columns to 'dc' * Add function to get latest encoding and use it in tests * Fix name DedicateColumnsFromTempopb * [vParquet3] Address more review comments * Remove 'Test' columns from vParquet3 schema * Rename async iterator environment variable * Do not export methods of dedicatedColumnMapping * Skip dedicated attribute lookup depending on scope in searchTagValues * Validate maximum number of configured dedicated columns * Test data for vparquet3 uses dedicated columns * Reduce size of block meta JSON * Use 'parquet_' prefix for dedicated column configuration * [vParquet3] Integration tests with dedicated attribute columns * Add e2e tests for encodings and dedicated attribute columns * Use dedicated attribute columns in TestSearchCompleteBlock * Add support for v2 in encodings test --------- Co-authored-by: Mario <[email protected]>
2 lines
511 B
JSON
2 lines
511 B
JSON
{"format":"vParquet3","blockID":"b27b0e53-66a0-4505-afd6-434ae3cd4a10","minID":"AAAAAAAAAAAAR0votDRJ+w==","maxID":"AAAAAAAAAAD/+S7r9o+CMA==","tenantID":"single-tenant","startTime":"2022-07-04T11:11:09Z","endTime":"2022-07-04T11:11:35Z","totalObjects":134,"size":65263,"compactionLevel":0,"encoding":"none","indexPageSize":0,"totalRecords":1,"dataEncoding":"","bloomShards":1,"footerSize":12442,"dedicatedColumns":[{"s":"resource","n":"ip"},{"n":"instance"},{"n":"version"},{"n":"region"},{"n":"sampler.type"}]}
|