checkpoint image compression (compressio) implies additional overhead
during its operations. when gvisor restores the kernel state inflate()
algorithm requires:
- CPU to un/compress the data
- Memory blocks to store and un/compress data
memory blocks originate from the bytes.Buffers and the sync.Pool that
tries to reuse them. they are released only when the system decides it
is a good moment:
pool.go: runtime_registerPoolCleanup(poolCleanup)
in my system (and in production) it takes around 240s to get the related
memory region freed (unmapped()).
during that period of time from the image state is read and the kernel is
loaded till the moment when the `poolCleanup` is called + GC() releasing
buffers gVisor Kernel (sandbox) process holds tens and hundreds of
megabytes of anonymous memory pages (RAM) busy (allocated+reserved).
pretty much often, the memory overhead of using compression can result in x2
memory overhead in production system with checkpoints restore and +100ms
(hundreds) ms of startup latency just to uncompress the image.
our use case does not suffer from having uncompressed images on disk but
suffer from the waste of memory during startup and CPU overhead.
this patch adds flag to disable compression for containers checkpoints.
Signed-off-by: Ivan Prisyazhnyy <[email protected]>
This is an effort to reduce it to be a well-behaved background process.
With 110 sandboxes running, at rest, this goes from
```
VmRSS: 72376 kB
RssAnon: 51944 kB
```
to:
```
VmRSS: 45864 kB
RssAnon: 25788 kB
```
This GCs much more aggressively, including after every single request, which
means we do spend disproportionately more CPU in order to get that low memory
usage. From my testing, serving requests takes about 12% more CPU, and it's
all spent in GC.
The optimizations that went into this are:
- Add a method in `state` to discard the global type maps.
- Add a custom "packed" number type in `prometheus` library that encodes small
integers and floating-point numbers in 32 bits whenever possible without
loss of precision, otherwise they are encoded in their full 64-bit glory and
the 32-bit representation is used as a pointer to the 64-bit representation.
These are stored either per-sandbox (for static-after-sandbox-creation
numbers like distribution bucket boundaries), or per-metric-retrieval
attempt otherwise.
- Use string interning for commonly-seen strings across sandboxes, like metric
names and label names. Label values are also interned, but only at a
per-sandbox granularity.
- Reworked allocation-heavy functions like `OrderedLabels` and some string
rendering functions to be (almost) allocation-free. This doesn't reduce
memory usage at rest, and does increase their CPU cost, but in return it
significantly cuts down on the percentage of CPU time spent in GC
(>50% -> 25%) enough to justify spending the extra CPU in these functions.
PiperOrigin-RevId: 515181387
This removes the need for ongoing tags.
This change requires some minor updates to remove dependency cycles, since
the goid package is a base library used by many internals (log, sync, etc.).
PiperOrigin-RevId: 504066914
fs.syncableDentries saves all non-synthetic dentries. This requires a map insert
operation every time a new dentry is created and map removal operation when a
dentry is destroyed. This can be expensive there can be a very large number of
non-synthetic dentries.
Using a map does not provide any additional benefits. We do not require lookup.
Instead use a linked list, as insert and remove are really fast and it allows us
to iterate on the list. It also saves the heap allocations to maintain the map.
Also simplify pkg/state to not use a custom ElementMapper. There is no need to.
PiperOrigin-RevId: 476145150
This adds significant costs to startup, since it is done for
every type in the system. Since the state package already saves
sanity checks for race builds, use this for type registration.
PiperOrigin-RevId: 350259336
- When encodeState.resolve() determines that the resolved reflect.Value is
contained by a previously-resolved object, set wire.Ref.Type to the
containing object's type (existing.obj.Type()) rather than the contained
value's type (obj.Type()).
- When encodeState.resolve() determines that the resolved reflect.Value
contains a previously-resolved object, handle cases where the new object
contains *multiple* previously-resolved objects. (This may cause
previously-allocated object IDs to become unused; to facilitate this, change
encodeState.pending to a map, and change the wire format to prefix each
object with its object ID.)
- Add encodeState.encodedStructs to avoid redundant encoding of structs, since
deduplication of objects via encodeState.resolve() doesn't work for objects
instantiated by StateSave() and passed to SaveValue() (i.e. fields tagged
`state:".(whatever)"`).
- Make unexported array fields deserializable via slices that refer to them by
casting away their unexportedness in decodeState.decodeObject().
Updates #1663
PiperOrigin-RevId: 338727687
Previously, it was not possible to encode/decode an object graph which
contained a pointer to a field within another type. This was because the
encoder was previously unable to disambiguate a pointer to an object and a
pointer within the object.
This CL remedies this by constructing an address map tracking the full memory
range object occupy. The encoded Refvalue message has been extended to allow
references to children objects within another object. Because the encoding
process may learn about object structure over time, we cannot encode any
objects under the entire graph has been generated.
This CL also updates the state package to use standard interfaces intead of
reflection-based dispatch in order to improve performance overall. This
includes a custom wire protocol to significantly reduce the number of
allocations and take advantage of structure packing.
As part of these changes, there are a small number of minor changes in other
places of the code base:
* The lists used during encoding are changed to use intrusive lists with the
objectEncodeState directly, which required that the ilist Len() method is
updated to work properly with the ElementMapper mechanism.
* A bug is fixed in the list code wherein Remove() called on an element that is
already removed can corrupt the list (removing the element if there's only a
single element). Now the behavior is correct.
* Standard error wrapping is introduced.
* Compressio was updated to implement the new wire.Reader and wire.Writer
inteface methods directly. The lack of a ReadByte and WriteByte caused issues
not due to interface dispatch, but because underlying slices for a Read or
Write call through an interface would always escape to the heap!
* Statify has been updated to support the new APIs.
See README.md for a description of how the new mechanism works.
PiperOrigin-RevId: 318010298
These packages don't actually use go_stateify or go_marshal, but end
up implicitly dependent on the respective packages due to our build
rules.
These unnecessary dependencies make them unusuable in certain contexts
due to circular dependency.
PiperOrigin-RevId: 312595738
Addresses obvious typos, in the documentation only.
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/gvisor/pull/443 from Pixep:fix/documentation-spelling 4d0688164eafaf0b3010e5f4824b35d1e7176d65
PiperOrigin-RevId: 255477779
Based on the guidelines at
https://opensource.google.com/docs/releasing/authors/.
1. $ rg -l "Google LLC" | xargs sed -i 's/Google LLC.*/The gVisor Authors./'
2. Manual fixup of "Google Inc" references.
3. Add AUTHORS file. Authors may request to be added to this file.
4. Point netstack AUTHORS to gVisor AUTHORS. Drop CONTRIBUTORS.
Fixes#209
PiperOrigin-RevId: 245823212
Change-Id: I64530b24ad021a7d683137459cafc510f5ee1de9
Nothing reads them and they can simply get stale.
Generated with:
$ sed -i "s/licenses(\(.*\)).*/licenses(\1)/" **/BUILD
PiperOrigin-RevId: 231818945
Change-Id: Ibc3f9838546b7e94f13f217060d31f4ada9d4bf0
This change also adds extensive testing to the p9 package via mocks. The sanity
checks and type checks are moved from the gofer into the core package, where
they can be more easily validated.
PiperOrigin-RevId: 218296768
Change-Id: I4fc3c326e7bf1e0e140a454cbacbcc6fd617ab55
Compared to previous compressio / hashio nesting, there is up to 100% speedup.
PiperOrigin-RevId: 210161269
Change-Id: I481aa9fe980bb817fe465fe34d32ea33fc8abf1c
We have been unnecessarily creating too many savable types implicitly.
PiperOrigin-RevId: 206334201
Change-Id: Idc5a3a14bfb7ee125c4f2bb2b1c53164e46f29a8
Ivan Prisyazhnyy