Update the memmap IncRef method to pass memory cgroup id and store it in the
FrameRefSet which will be used for memory accounting. During DecRef, the
memCgID from the FrameRefSet will be retrieved and passed to MemoryLocked.Dec
to remove the memory from the cgroup.
PiperOrigin-RevId: 549656411
- Adds a new field in the usageInfo to store the memory cgroup id.
- Creates a map of cgroup ids and memory stats to track the memory per cgroup
in MemoryLocked struct.
- Introduces new methods to increment, decrement, move, copy and get the total
memory usage per cgroup.
PiperOrigin-RevId: 549148091
Earlier we had another method for AllocateAndPopulate. This change moves the
additional (optional) parameters to pgalloc.AllocOpts.
* Makes `Reader` parameter optional. Not all callers have data to fill into
the allocated pages. Avoids having to define no-op readers.
* Avoid calling MapInternal() when `Reader == nil` and
`Mode != AllocateAndWritePopulate`.
* Avoid calling safemem.ReadFullToBlocks() when `Reader == nil`.
* Move AllocationMode upgrade/downgrade logic to callsites. This makes
Allocate() simpler.
PiperOrigin-RevId: 547688619
For memory accounting per task, the memory cgroup id of the task is required.
This CL retrieves the cgroup id from the task context and stores it in the
AllocOpts struct which will later be used to account for memory usage.
PiperOrigin-RevId: 546996195
When an application makes a huge write, it is better to pre-fault all the pages
that will later be touched by runtime.memmove() via safemem.CopySeq(). Earlier
we were calling MemoryFile.Allocate(), which does not populate the pages.
Otherwise, these pages fault one at a time as they are touched, which is really
slow due to context switching.
Note that this optimization only works for shmem-backed MemoryFiles. This means
that writes to unallocated regions of tmpfs mounts (like /tmp) will be faster.
rootfs (overlay) does not benefit because even though it has an upper tmpfs
layer, the MemoryFile used is disk-backed.
We only prepopulate if more than 1 page is being written to. This is because
the syscall path is susceptible to certain overheads that the page fault path
does not have. For example seccomp-bpf and syscall auditing.
The improved large writes in tmpfs is evidenced by (on Linux >=5.14):
- Before
```
$ docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 3.10212 s, 1.3 GB/s
```
- After
```
docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 2.38347 s, 1.7 GB/s
```
In this workload, this change reduces latency by 23% (3.10212s -> 2.38347s).
PiperOrigin-RevId: 544056038
Pre-populating pages on fallocate(2) when the MemoryFile is backed by a disk
is wasteful.
It is currently wasteful because the mlock+munlock mechanism used in
pgalloc.tryPopulate() to populate pages does not work with some disk-backed
filesystems like ext4. We end up faulting read-only pages, which need to
be re-faulted on write anyways.
Later we will update pgalloc.tryPopulate() to use madvice(MADV_POPULATE_WRITE).
It wouldn't work in that case too because even though it does fault writable
pages, they are written back to disk before the application can write to them.
And so they will be re-faulted on write. This would actually worsen the
performance because now we have the extra cost of writeback.
Hence, avoid populating pages for disk-backed MemoryFile on fallocate(2).
Instead we just commit the pages to satisfy fallocate(2) semantics.
When the application writes to these file offsets, the pages will fault
normally.
To allow this distinction between "committing" and "populating" pages in
addition to allocating them, the pgalloc API has been extended by introducing
`pgalloc.AllocationMode` enum. It allows the caller to select if pages should
be only-allocated, allocated+committed or allocated+committed+populated.
Here are the benchmarking results:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ sec/op │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-4 100.73 ± 12% 99.72 ± 58% ~ (p=0.805 n=7)
BuildGRPC/page_cache.clean/filesystem.bindfs-4 703.5 ± 1% 690.9 ± 1% -1.78% (p=0.004 n=7)
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 2.008m ± 0% 2.009m ± 25% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=0.318 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 2.510m ± 0% 2.509m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 2.009m ± 0% 2.009m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.097 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 3.010m ± 17% 2.510m ± 0% ~ (p=0.053 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 2.009m ± 25% 2.008m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.259 n=7)
RubySpecTest/page_cache.clean/filesystem.bindfs-4 63.15 ± 2% 61.65 ± 2% -2.38% (p=0.011 n=7)
geomean 34.15m 33.48m -1.94%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ bandwidth.bytes_per_second │ bandwidth.bytes_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 939.6M ± 2% 944.7M ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 2.937G ± 5% 2.873G ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 3.139G ± 7% 3.112G ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 1.355G ± 3% 1.382G ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 3.913G ± 10% 3.785G ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.692G ± 11% 3.855G ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 788.4M ± 2% 628.6M ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 1346.1M ± 5% 940.4M ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.646G ± 5% 1.173G ± 3% -28.75% (p=0.001 n=7)
geomean 1.882G 1.699G -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ io_ops.ops_per_second │ io_ops.ops_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 229.4k ± 2% 230.6k ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 44.82k ± 5% 43.84k ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 2.994k ± 7% 2.967k ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 330.7k ± 3% 337.3k ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 59.70k ± 10% 57.76k ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.521k ± 11% 3.676k ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 192.5k ± 2% 153.5k ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 20.54k ± 5% 14.35k ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.570k ± 5% 1.119k ± 3% -28.73% (p=0.001 n=7)
geomean 28.72k 25.93k -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ load.sec │ load.sec vs base │
RubySpecTest/page_cache.clean/filesystem.bindfs-4 9.430 ± 7% 9.150 ± 7% -2.97% (p=0.026 n=7)
```
The broader filesystem benchmarks (ABSL, gRPC, Ruby) all improved with this
change. Improvement is ~2-3% reduction in absolute time to run these
workloads. However, the FIO microbenchmark is a little more complicated.
The latency metric shows improvement (for blockSize=4K & filesystem=rootfs).
It is stable for other blockSize values. But FIO bandwidth seems to have
regressed. We think this is an accounting gimmick. We suspect that sec/op
includes the app fallocate() and bandwidth does not. Since this change moves
ext4_mpage_readpages() from app fallocate() to the page faults that occur
during app write(), bandwidth appears to have regressed.
Given that the broader filesystem benchmarks look good and that this is more
consistent with Linux (app fallocate() now only commits disk space rather than
also materializing memory pages), this change should be an improvement.
PiperOrigin-RevId: 542331863
Earlier we were using the same filestore for all containers in the
sandbox. This was based on the assumption that all containers in a
k8s pod have the same runsc configuration. We get rid of our
dependence on that assumption. Subcontainers can independently use
overlay configurations.
This is needed because in k8s, different ephemeral storage limits can
be enforced on containers in the same pod. This behavior is required
to get limit enforcement to work correctly on a per container basis.
This change also moves the ownership of overlay filestore to tmpfs.
Now that each container has its own overlay-backing MemoryFile, it will
be much harder to manage lifecycle of these MemoryFiles as containers
are created and destroyed.
Instead, move ownership to tmpfs mount. When the overlay upper layer tmpfs
is unmounted, it will destroy the MemoryFile.
We also extract out IMA work around logic from NewMemoryFile() and perform
it outside the sandbox. This is done because now tmpfs is creating the
MemoryFile. By then the sandbox is already running with seccomp filters
installed that do not allow certain mmap(2) calls that the said work around
uses. We hit the same issue for subcontainers. When they are started,
the sandbox is already running with seccomp filters. Added a new
MemoryFileOpts field for this which defaults to old behavior.
PiperOrigin-RevId: 505892085
MemoryFile.findReclaimable() immediately exits when MemoryFile.Destroy() is
called, even if MemoryFile.reclaim set is not empty. If the MemoryFile is
backed by a host filesystem based file, then just closing MemoryFile.FD() does
not release resources held by the MemoryFile. To release the disk space held by
the host file, it needs to be decommitted. Otherwise, we can leave around large
host files.
Guard this change in behavior behind a flag in MemoryFileOpts which defaults to
the old behavior of not decommitting the host file.
PiperOrigin-RevId: 505884271
--overlay2 flag supersedes --overlay flag. It allows more granular
configuration for overlayfs in runsc. It does so in two ways:
1. Allows to apply overlay on all mounts or only the root mount.
--overlay applies overlay to all mounts.
2. Allows to specify if overlay's upper layer should be backed by
container memory or disk. --overlay always used container memory.
Allowing tmpfs to be backed by a file on disk prevents the container
memory from bloating up. Note that the tmpfs filesystem tree will
still be stored in sentry memory.
Using overlay on the root filesystem, helps avoid expensive
communication with the gofer process. The root filesystem of the
container is not preserved across container lifecycle. So we don't
need to keep updating the host filesystem, which will anyways be
destroyed once the container is destroyed. It is wasted effort.
Instead we keep all the changes to the root filesystem in tmpfs which
is directly accessible by the sentry.
The host file is created as an unnamed file using O_TMPFILE. Support
has been added for sub-containers too. Save/restore support is
still lacking.
Co-authored-by: Andrei Vagin <[email protected]>
PiperOrigin-RevId: 491988485
MemoryFile allocated offsets from the top down. This can generate
fragmentation when allocations happen in the opposite direction.
Given that applications control the order of allocations, use simple
heuristics to track the last faulted address to determine whether memory
file should be allocated from the top-down or bottom-up.
Here is the number of PMAs used across all processes in some common
workloads:
Workload | Before | After | Diff
---------|---------|---------|------
rustc | 135,269 | 1,090 | 0.8%
mysql | 16,466 | 4,576 | 28%
nginx | 1,738 | 1,609 | 93%
jenkins | 13,830 | 6,380 | 46%
redis | 334 | 308 | 92%
absl | 297,419 | 292,121 | 98%
Here is the direction of PMA allocations:
Workload | BottomUp | TopDown |
---------|-----------|---------|
mysql | 13,849 | 1,990 |
nginx | 783 | 628 |
jenkins | 13,922 | 2,333 |
redis | 88 | 109 |
absl | 212,403 | 44,019 |
Tests were done on Intel x64.
PiperOrigin-RevId: 409014690
Removes package syserror and moves still relevant code to either linuxerr
or to syserr (to be later removed).
Internal errors are converted from random types to *errors.Error types used
in linuxerr. Internal errors are in linuxerr/internal.go.
PiperOrigin-RevId: 390724202
This change makes the checklocks analyzer considerable more powerful, adding:
* The ability to traverse complex structures, e.g. to have multiple nested
fields as part of the annotation.
* The ability to resolve simple anonymous functions and closures, and perform
lock analysis across these invocations. This does not apply to closures that
are passed elsewhere, since it is not possible to know the context in which
they might be invoked.
* The ability to annotate return values in addition to receivers and other
parameters, with the same complex structures noted above.
* Ignoring locking semantics for "fresh" objects, i.e. objects that are
allocated in the local frame (typically a new-style function).
* Sanity checking of locking state across block transitions and returns, to
ensure that no unexpected locks are held.
Note that initially, most of these findings are excluded by a comprehensive
nogo.yaml. The findings that are included are fundamental lock violations.
The changes here should be relatively low risk, minor refactorings to either
include necessary annotations to simplify the code structure (in general
removing closures in favor of methods) so that the analyzer can be easily
track the lock state.
This change additional includes two changes to nogo itself:
* Sanity checking of all types to ensure that the binary and ast-derived
types have a consistent objectpath, to prevent the bug above from occurring
silently (and causing much confusion). This also requires a trick in
order to ensure that serialized facts are consumable downstream. This can
be removed with https://go-review.googlesource.com/c/tools/+/331789 merged.
* A minor refactoring to isolation the objdump settings in its own package.
This was originally used to implement the sanity check above, but this
information is now being passed another way. The minor refactor is preserved
however, since it cleans up the code slightly and is minimal risk.
PiperOrigin-RevId: 382613300
Update all instances of the above errors to the faster linuxerr implementation.
With the temporary linuxerr.Equals(), no logical changes are made.
PiperOrigin-RevId: 382306655
Split usermem package to help remove syserror dependency in go_marshal.
New hostarch package contains code not dependent on syserror.
PiperOrigin-RevId: 365651233
This validates that struct fields if annotated with "// checklocks:mu" where
"mu" is a mutex field in the same struct then access to the field is only
done with "mu" locked.
All types that are guarded by a mutex must be annotated with
// +checklocks:<mutex field name>
For more details please refer to README.md.
PiperOrigin-RevId: 360729328
The syscall package has been deprecated in favor of golang.org/x/sys.
Note that syscall is still used in the following places:
- pkg/sentry/socket/hostinet/stack.go: some netlink related functionalities
are not yet available in golang.org/x/sys.
- syscall.Stat_t is still used in some places because os.FileInfo.Sys() still
returns it and not unix.Stat_t.
Updates #214
PiperOrigin-RevId: 360701387
This change makes the following changes:
- Unlocks MemoryFile.mu while calling mincore (checkCommitted) because mincore
can take a really long time. Accordingly looks up the segment in the tree
tree again and handles changes to the segment.
- MemoryFile.UpdateUsage() can now only be called at frequency at most 100Hz.
100 Hz = linux.CLOCKS_PER_SEC.
Co-authored-by: Jamie Liu <[email protected]>
PiperOrigin-RevId: 337865250
Our "Preconditions:" blocks are very useful to determine the input invariants,
but they are bit inconsistent throughout the codebase, which makes them harder
to read (particularly cases with 5+ conditions in a single paragraph).
I've reformatted all of the cases to fit in simple rules:
1. Cases with a single condition are placed on a single line.
2. Cases with multiple conditions are placed in a bulleted list.
This format has been added to the style guide.
I've also mentioned "Postconditions:", though those are much less frequently
used, and all uses already match this style.
PiperOrigin-RevId: 327687465
Previously, it was not possible to encode/decode an object graph which
contained a pointer to a field within another type. This was because the
encoder was previously unable to disambiguate a pointer to an object and a
pointer within the object.
This CL remedies this by constructing an address map tracking the full memory
range object occupy. The encoded Refvalue message has been extended to allow
references to children objects within another object. Because the encoding
process may learn about object structure over time, we cannot encode any
objects under the entire graph has been generated.
This CL also updates the state package to use standard interfaces intead of
reflection-based dispatch in order to improve performance overall. This
includes a custom wire protocol to significantly reduce the number of
allocations and take advantage of structure packing.
As part of these changes, there are a small number of minor changes in other
places of the code base:
* The lists used during encoding are changed to use intrusive lists with the
objectEncodeState directly, which required that the ilist Len() method is
updated to work properly with the ElementMapper mechanism.
* A bug is fixed in the list code wherein Remove() called on an element that is
already removed can corrupt the list (removing the element if there's only a
single element). Now the behavior is correct.
* Standard error wrapping is introduced.
* Compressio was updated to implement the new wire.Reader and wire.Writer
inteface methods directly. The lack of a ReadByte and WriteByte caused issues
not due to interface dispatch, but because underlying slices for a Read or
Write call through an interface would always escape to the heap!
* Statify has been updated to support the new APIs.
See README.md for a description of how the new mechanism works.
PiperOrigin-RevId: 318010298
findAvailableLocked() may return a non-aligned FileRange.End after expansion
since it may round FileRange.Start down to a hugepage boundary.
PiperOrigin-RevId: 315520321
This change has multiple small components.
First, the chunk size is bumped to 1GB in order to avoid creating excessive
VMAs in the Sentry, which can lead to VMA exhaustion (and hitting limits).
Second, gap-tracking is added to the usage set in order to efficiently scan
for available regions.
Third, reclaim is moved to a simple segment set. This is done to allow the
order of reclaim to align with the Allocate order (which becomes much more
complex when trying to track a "max page" as opposed to "min page", so we
just track explicit segments instead, which should make reclaim scanning
faster anyways).
Finally, the findAvailable function attempts to scan from the top-down, in
order to maximize opportunities for VMA merging in applications (hopefully
preventing the same VMA exhaustion that can affect the Sentry).
PiperOrigin-RevId: 315009249
Because the abi will depend on the core types for marshalling (usermem,
context, safemem, safecopy), these need to be flattened from the sentry
directory. These packages contain no sentry-specific details.
PiperOrigin-RevId: 291811289
* Rename syncutil to sync.
* Add aliases to sync types.
* Replace existing usage of standard library sync package.
This will make it easier to swap out synchronization primitives. For example,
this will allow us to use primitives from github.com/sasha-s/go-deadlock to
check for lock ordering violations.
Updates #1472
PiperOrigin-RevId: 289033387
This feature allows MemoryFile to delay eviction of "optional"
allocations, such as unused cached file pages.
Note that this incidentally makes CachingInodeOperations writeback
asynchronous, in the sense that it doesn't occur until eviction; this is
necessary because between when a cached page becomes evictable and when
it's evicted, file writes (via CachingInodeOperations.Write) may dirty
the page.
As currently implemented, this feature won't meaningfully impact
steady-state memory usage or caching; the reclaimer goroutine will
schedule eviction as soon as it runs out of other work to do. Future CLs
increase caching by adding constraints on when eviction is scheduled.
PiperOrigin-RevId: 246014822
Change-Id: Ia85feb25a2de92a48359eb84434b6ec6f9bea2cb
Based on the guidelines at
https://opensource.google.com/docs/releasing/authors/.
1. $ rg -l "Google LLC" | xargs sed -i 's/Google LLC.*/The gVisor Authors./'
2. Manual fixup of "Google Inc" references.
3. Add AUTHORS file. Authors may request to be added to this file.
4. Point netstack AUTHORS to gVisor AUTHORS. Drop CONTRIBUTORS.
Fixes#209
PiperOrigin-RevId: 245823212
Change-Id: I64530b24ad021a7d683137459cafc510f5ee1de9
This is in preparation for improved page cache reclaim, which requires
greater integration between the page cache and page allocator.
PiperOrigin-RevId: 238444706
Change-Id: Id24141b3678d96c7d7dc24baddd9be555bffafe4
