mirror of
https://github.com/hanzoai/traces.git
synced 2026-10-30 18:20:41 +00:00
049323c40f
* starting parquet support starting parquet support. some things are still todo: compaction, find trace by ID Co-Authored-By: Annanay Agarwal <[email protected]> Co-Authored-By: Martin Disibio <[email protected]> * add parquet reusable streaming block * Add parquet compaction * Add findTraceByID logic and parquet -> proto conversion Signed-off-by: Annanay <[email protected]> * Switch findtracebyid to readrow until we get SeekToRow working * Fix error handling in bufferedReaderAt. Add various instrumentation and log messages * Use jsonpb to roundtrip event attributes * Improve buffered reader when reading near end of file. More instrumentation and tidiness * Add totalBlockBytes instrumentation. Fix inspectedBytes instrumentation * parquet.FindTraceByID check bloom * Make compator flush on trace count until we get the row group size target working * Support maxDuration in parquet search * Correct attribute translation for array/kvlist Signed-off-by: Annanay <[email protected]> * extend roundtrip test Signed-off-by: Annanay <[email protected]> * Use default page buffer size * Fix size url param to be uint64 * Increase parquet searches to 1MB stripes * Add test for findtracebyid * make row group size configurable Signed-off-by: Annanay <[email protected]> * Record parquet footer size in block meta and short-circuit header/footer reads * Add tests for SubstringPredicate * Switch compaction to compact blocks of same version, use re-usable buffer in findtracebyid Signed-off-by: Annanay <[email protected]> * Add search support for remaining well-known attributes and span status, add search tests * Benchmark substringPredicate * Make block version configurable, default to v2 * memoize substringPredicate * Update segmentio/parquet-go to catch SeekToRow fix, revert hack in FindTraceByID Signed-off-by: Annanay <[email protected]> * Add multi block prefetch iterator for use in compactor Signed-off-by: Annanay <[email protected]> * Add BufferedWriter with outgoing queue to flush traces async Signed-off-by: Annanay <[email protected]> Make compactor block complete async Signed-off-by: Annanay <[email protected]> * First pass CLI to dump schema and column sizes from block Signed-off-by: Annanay <[email protected]> * honor search start/end time range * Improve error message, add blockID Signed-off-by: Annanay <[email protected]> * Fix race in InstrumentedPredicate with atomics * Make bufferedReaderAt not single-threaded * Debug SeekToRow panic, add test data from vulture tenant Signed-off-by: Annanay <[email protected]> * fix test * repair tests * Address println warnings * Experiment with bookmark prefetch Signed-off-by: Annanay <[email protected]> Remove quitCh from bookmark Signed-off-by: Annanay <[email protected]> * fix prefetch on the bookmarks Signed-off-by: Annanay <[email protected]> * lint * tracker -> RowNumber, lint * delete unused predicates * lint * lint * mod vendor * remove pool from buffered writer queue Signed-off-by: Annanay <[email protected]> * post merge cleanup Signed-off-by: Annanay <[email protected]> * Close quitCh instead of attempting to signal to it Signed-off-by: Annanay <[email protected]> * split multiblock and prefetch iterators, refactor around interface Signed-off-by: Annanay <[email protected]> * Split search into 2 phases to avoid loading traceid/display columns unless there is a match * Make columnIterator.SeekTo do something * update to catch segmentio/parquet-go#220 Signed-off-by: Annanay <[email protected]> * Fix column index to store entire 32-byte trace ID string. lint * Make parquet honor the adjusted start/end times that come from the wal * post merge cleanup: use compaction metric callbacks Signed-off-by: Annanay <[email protected]> * serverless sum/mods Signed-off-by: Annanay <[email protected]> * parquetquery predicates handle column index not present * Make parquet compaction really honor input block start/end times, add universal test at tempodb-level * parquet skip page indexes while searching to reduce latency on openfile * More testing around iterators and combiner Signed-off-by: Annanay <[email protected]> * Update to latest parquet lib to catch rle bitpack fix * Review feedback: LatestEncoding -> DefaultEncoding * Pass block footerSize through serverless * mod vendor for serverless * Fix tests * add blockID to error message in block iterator Signed-off-by: Annanay <[email protected]> * increase readvalues buffer, add more span instrumentation Signed-off-by: Annanay <[email protected]> * changelog and docs * Remove the use of async writer in vparquet compactor Signed-off-by: Annanay <[email protected]> Co-authored-by: Annanay Agarwal <[email protected]>
47 lines
2.3 KiB
Go
47 lines
2.3 KiB
Go
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style
|
|
// license that can be found in the LICENSE file.
|
|
|
|
// Package snappy implements the Snappy compression format. It aims for very
|
|
// high speeds and reasonable compression.
|
|
//
|
|
// There are actually two Snappy formats: block and stream. They are related,
|
|
// but different: trying to decompress block-compressed data as a Snappy stream
|
|
// will fail, and vice versa. The block format is the Decode and Encode
|
|
// functions and the stream format is the Reader and Writer types.
|
|
//
|
|
// The block format, the more common case, is used when the complete size (the
|
|
// number of bytes) of the original data is known upfront, at the time
|
|
// compression starts. The stream format, also known as the framing format, is
|
|
// for when that isn't always true.
|
|
//
|
|
// The canonical, C++ implementation is at https://github.com/google/snappy and
|
|
// it only implements the block format.
|
|
package snappy
|
|
|
|
/*
|
|
Each encoded block begins with the varint-encoded length of the decoded data,
|
|
followed by a sequence of chunks. Chunks begin and end on byte boundaries. The
|
|
first byte of each chunk is broken into its 2 least and 6 most significant bits
|
|
called l and m: l ranges in [0, 4) and m ranges in [0, 64). l is the chunk tag.
|
|
Zero means a literal tag. All other values mean a copy tag.
|
|
|
|
For literal tags:
|
|
- If m < 60, the next 1 + m bytes are literal bytes.
|
|
- Otherwise, let n be the little-endian unsigned integer denoted by the next
|
|
m - 59 bytes. The next 1 + n bytes after that are literal bytes.
|
|
|
|
For copy tags, length bytes are copied from offset bytes ago, in the style of
|
|
Lempel-Ziv compression algorithms. In particular:
|
|
- For l == 1, the offset ranges in [0, 1<<11) and the length in [4, 12).
|
|
The length is 4 + the low 3 bits of m. The high 3 bits of m form bits 8-10
|
|
of the offset. The next byte is bits 0-7 of the offset.
|
|
- For l == 2, the offset ranges in [0, 1<<16) and the length in [1, 65).
|
|
The length is 1 + m. The offset is the little-endian unsigned integer
|
|
denoted by the next 2 bytes.
|
|
- For l == 3, this tag is a legacy format that is no longer issued by most
|
|
encoders. Nonetheless, the offset ranges in [0, 1<<32) and the length in
|
|
[1, 65). The length is 1 + m. The offset is the little-endian unsigned
|
|
integer denoted by the next 4 bytes.
|
|
*/
|