mirror of
https://github.com/luxfi/zapdb.git
synced 2026-10-30 15:12:20 +00:00
Ibrahim Jarif
58a11fabb6
This commit adds support for compression/encryption in the background.
All incoming writes are saved to a common buffer and then worker go
routine pick up each block (a block is just a start and end point in
the main buffer). The worker goroutines also fix the table start and
end points. We fix the interleaving space (which might be caused because
the block was compressed) in the builder.Finish() call. All the blocks
are copied over to their correct position.
Master branch vs This commit (new) Benchmarks.
```
benchstat master.txt new.txt
name old time/op new time/op delta (-ve is better. Reduction in time)
no_compression-16 177ms ± 1% 177ms ± 2% ~ (p=0.690 n=5+5)
encryption-16 313ms ± 4% 245ms ± 3% -21.69% (p=0.008 n=5+5)
zstd_compression/level_1-16 330ms ± 1% 227ms ± 2% -31.22% (p=0.008 n=5+5)
zstd_compression/level_3-16 345ms ± 2% 227ms ± 3% -34.39% (p=0.008 n=5+5)
zstd_compression/level_15-16 10.7s ± 1% 1.2s ± 0% -88.96% (p=0.008 n=5+5)
name old speed new speed delta (+ve is better. Speed improvement)
no_compression-16 471MB/s ± 1% 471MB/s ± 2% ~ (p=0.690 n=5+5)
encryption-16 266MB/s ± 4% 340MB/s ± 2% +27.67% (p=0.008 n=5+5)
zstd_compression/level_1-16 252MB/s ± 1% 367MB/s ± 2% +45.40% (p=0.008 n=5+5)
zstd_compression/level_3-16 241MB/s ± 2% 367MB/s ± 3% +52.44% (p=0.008 n=5+5)
zstd_compression/level_15-16 7.80MB/s ± 1% 70.62MB/s ± 0% +805.33% (p=0.008 n=5+5)
```
Please look at https://github.com/dgraph-io/badger/pull/1227#issue-378982789
for the code used for benchmarking.
(cherry picked from commit b13b927102)