MemoryManager.Pin() is used by device proxies (nvproxy, tpuproxy,
rdmaproxy) to pin application pages for DMA, analogous to Linux's
pin_user_pages(FOLL_LONGTERM). Fork() previously made all private pmas
copy-on-write, including pinned ones. The parent's next write to a
pinned page would break copy-on-write and move the parent to a new copy
of the page, while the device continues to DMA to/from the original
page, silently desynchronizing the two. Linux fixed the same bug in
5.9; see Linux commit 70e806e4e645 ("mm: Do early cow for pinned pages
during fork() for ptes").
Match Linux:
- Pin() now breaks copy-on-write on the pinned range, even for
read-only pins, so that pinned pages are exclusively owned by the
pinning MemoryManager; compare Linux's
mm/internal.h:gup_must_unshare(). This also makes pinned pages
precisely detectable at fork time: they are exactly the pages in
private non-copy-on-write pmas with more than one reference on the
MemoryFile.
- Fork() copies possibly-pinned pages to the child immediately, leaving
the parent's mappings unchanged, instead of making them
copy-on-write; compare Linux's mm/memory.c:copy_present_ptes() =>
folio_needs_cow_for_dma(). Detection is gated on the new
MemoryManager.hasPinned, set by Pin() and never cleared; compare
Linux's MMF_HAS_PINNED.
MemoryFile.UpdateUsage() detects newly-committed pages by asking the
host, via mincore(), about every page not already known to be committed.
This is O(uncommitted pages); in a sandbox with hundreds of GB of memory
under allocation churn, where pages are constantly decommitted and
recommitted, a single scan takes seconds.
Scans were throttled to CONFIG_HZ, which assumes that they are cheap.
Worse, the 10ms deadline was armed before the scan rather than after it,
so a scan exceeding 10ms did not prevent concurrent callers from starting
their own redundant scans.
Both are easy to hit, since reads of /proc/meminfo call UpdateUsage() and
are commonly polled. In a 2-node 8xH200 training job with a ~100GB
resident set, where a Ray memory monitor and the trainer's per-step
logging both read /proc/meminfo periodically, the sentry spent 78% of its
CPU in UpdateUsage() -- 1.65 CPUs' worth, i.e. more than one scan running
at all times. Scanning also contends for the lock that page allocation
needs, so this slowed the application itself: NCCL weight-sync buckets
took ~10s instead of the ~0.23s they take under runc.
Arm the deadline after the scan instead, at commitScanIntervalRatio times
the duration of the scan just completed, bounding scans to 1/8th of a CPU
regardless of sandbox size, and skip scans that would duplicate one
already in progress. Both apply only to full scans (memCgIDs == nil); a
scan restricted to specific memory cgroups is much cheaper and runs
unthrottled, so per-cgroup statistics stay fresh.
Track the in-progress scan in an atomic checked before mu is locked, so
that callers arriving during a scan neither block on mu nor contend for
it with the scan and with page allocation. Such a caller goes on to read
usage.MemoryAccounting, which the ongoing scan updates incrementally
under a different lock, so it observes a partially updated snapshot
rather than a stale one; since a scan only ever increases the counters,
that snapshot lies between the pre-scan and post-scan values. Callers
could already observe a scan's intermediate state, as they read
usage.MemoryAccounting after UpdateUsage() has returned and released mu.
All callers report statistics (/proc/meminfo, cgroupfs memory files,
`runsc events`, `runsc usage`) and already tolerated both the previous
throttle and scans skipped during checkpointing.
... and use it everywhere in preference to its string. This is to be able to
reliably inspect pgalloc.MemoryFile.ResourceID().Path in cl/879147587.
PiperOrigin-RevId: 883274090
During checkpointing, MemoryFile.SaveTo() narrows the set of pages known by
memory accounting to be "committed" to those containing non-zero bytes, in
order to avoid saving zero pages and therefore bloating the checkpoint. In the
process of doing so, it needs to touch pages in order to determine whether they
contain non-zero bytes, and does so without holding MemoryFile.mu (in a
MemoryFile.updateUsageLocked() callback). Thus, a concurrent call to
MemoryFile.UpdateUsage() => MemoryFile.updateUsageLocked() can racily observe
that touched zero pages are committed (via mincore()) and mark them
known-committed accordingly, causing them to be unintentionally saved in the
checkpoint.
When SaveOpts.ExcludeCommittedZeroPages is set, MemoryFile.SaveTo() does not
decommit previously-known-committed zero pages, since doing so would cost time;
the motivation for decommitting zero pages is to avoid increasing (real, host)
memory usage during checkpointing, but previously-known-committed pages must
have been using memory even before being touched. However, this significantly
widens the race window described above, since any future call to
MemoryFile.UpdateUsage() will observe said pages to be committed and mark them
known-committed again, effectively negating SaveOpts.ExcludeCommittedZeroPages.
To fix this, inhibit MemoryFile.UpdateUsage() during MemoryFile.SaveTo(); in
essence, when MemoryFile.SaveTo() is in progress, it exclusively defines what
pages are known-committed.
PiperOrigin-RevId: 733876220
When a pages file is provided to `runsc restore`, reads from that file are
asynchronous (via statefile.AsyncReader) in order to maximize throughput.
However, all such reads must complete before Kernel.LoadFrom() returns, so
applications cannot execute before MemoryFile loading is complete. The main
objective of this CL is to allow reads to continue after Kernel.LoadFrom()
returns, allowing applications to execute while MemoryFile loading is still in
progress. This behavior is user-visible: it affects whether deleting the pages
file frees disk space immediately on POSIX filesystems, may affect whether
deletion is possible on non-POSIX filesystems, and prevents unmounting
regardless. Thus it is flag-guarded as `runsc restore --background`.
MemoryFile ranges that have yet to be loaded, but that are being waited-for by
applications, should be prioritized over ranges for which no application is
waiting. This requires that application requests for data (calls to
MemoryFile.(memmap.File).DataFD/MapInternal()) are able to determine which
ranges have not yet been loaded, request reads for such ranges with elevated
priority, and wait for only those reads to be completed; none of these are
supported by the existing statefile.AsyncReader.
Thus:
- Add //pkg/sentry/pgalloc/aio, which provides an async I/O API that is
designed to be easily implementable using a goroutine pool, Linux native AIO,
or io_uring, though only includes a goroutine pool implementation. (io_uring
is widely disabled due to security vulnerabilities. In my testing, Linux
native AIO is slower than the goroutine pool, but this may change with lower
GOMAXPROCS which needs further testing.)
- Move I/O scheduling into pgalloc: introduce an async page loader goroutine
that is started by MemoryFile.LoadFrom() when async page loading is requested
(implicitly, via the existence of a pages file), which is responsible for
driving submission of read requests and handling their completions.
PiperOrigin-RevId: 679321884
This CL addresses the following major issues:
- When an application releases memory to the sentry, the sentry unconditionally
releases that memory to the host, rather than allowing it to be reused for
future allocations, in order to ensure that new allocations are uniformly
decommitted (use no memory): cl/145016083. In most cases, this should have
relatively little performance impact; since releasing memory from the
application to the OS is expensive even outside of gVisor, application memory
allocators optimizing for performance already limit the rate at which they
release memory to the OS. However, in applications that involve frequent
process creation and exit (e.g. build systems), this practice prevents reuse
of memory deallocated by exiting processes for memory allocated by new
processes, resulting in both performance degradation and a spike in memory
usage (since the sentry may not have released all deallocated memory to the
host by the time new allocations occur).
- gVisor's historical approach to application THP is based on THP being enabled
on a per-memfd basis, using the MFD_HUGEPAGE flag not merged into the
upstream Linux kernel
(https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/).
Thus, on vanilla Linux kernels, gVisor cannot use THP for application memory
without requiring the system to enable THP for all tmpfs files and memfds (by
setting /sys/kernel/mm/transparent_hugepage/shmem_enabled to "always" or
"force").
- Both MM and the application page allocator (pgalloc) are agnostic as to
whether the underlying memory file will be THP-backed. Instead, both attempt
to align hugepage-sized and larger allocations to hugepage boundaries, such
that if the memory file happens to support THP then such allocations will be
appropriately aligned to use THP. This is suboptimal since many allocations
do not benefit from THP, resulting in memory underutilization.
These issues are especially relevant to platforms based on hardware
virtualization, where acquiring memory from the host is significantly more
expensive due to EPT/NPT fault overhead; when effective, THP reduces the
frequency with which said cost is incurred by a factor of 512, and page reuse
avoids incurring it at all.
Thus:
- Instead of inferring whether THP use is desired from allocation size,
indicate this explicitly as AllocOpts.Huge, and only set it to true for
allocations for non-stack private anonymous mappings.
- Add AllocateCallerIndirectCommit, a new possible value for AllocOpts.Mode
that indicates that the caller will commit all pages in the allocation. In
such cases, pgalloc can reuse deallocated pages without risking increased
memory usage, internally referred to as "recycling".
AllocateCallerIndirectCommit is used primarily for page faults on a
THP-backed region. (It is also used for single-page allocations on non-THP
backed regions, but due to expansion of faults to mm.privateAllocUnit-aligned
ranges, this is relatively uncommon.)
- Allow different chunks in pgalloc.MemoryFile's backing file to have varying
THP-ness, indicated to the host using MADV_HUGEPAGE/NOHUGEPAGE.
- Split pgalloc.MemoryFile's existing page metadata set into two sets tracking
deallocated pages for small/huge-page-backed regions respectively; two sets
tracking in-use pages for small/huge-page-backed regions respectively; and a
fifth set tracking memory accounting state.
- Add MemoryFileOpts.DisableMemoryAccounting; this is primarily intended for
pgalloc tests, but may also be applicable to disk-backed MemoryFiles.
Cleanup:
- Remove MemoryFile.usageSwapped; the UpdateUsage() optimization it enabled,
described in updateUsageLocked(), was based on the condition that
MemoryFile.mu would be locked throughout the call to updateUsageLocked(),
which was invalidated by cl/337865250.
- Remove MemoryFileOpts.ManualZeroing, which is unused.
- Rename "reclaiming" to "releasing"; the former is confusing since "reclaim"
in Linux has a significantly different meaning (essentially "eviction" in
pgalloc), and the latter seems to be conventional in user-mode memory
allocators.
Using THP for application memory requires setting
/sys/kernel/mm/transparent_hugepage/shmem_enabled to "advise", in order to
allow runsc to request THP from the kernel.
After this CL, pgalloc.MemoryFile still releases memory to the host as fast as
possible, limiting the effectiveness of page recycling. A following CL adds
optional memory release throttling to improve this.
Performance outcomes vary by workload and platform. (In all of the below,
"baseline" is without this CL, "expt" is with this CL, and "expt2" is with this
CL + reclaim throttling (cl/575046398).)
For systrap in GKE: As noted, this change is required to enable application THP
without forcing it on all host shmem users. In conjunction with recycling
(which has a relatively small effect on systrap since it does not use hardware
virtualization), THP use slightly improves performance, although whether this
is measurable is case-dependent. On an idle VM, with shmem_enabled = "advise":
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ baseline │ expt │ expt2 │
│ sec/op │ sec/op vs base │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-16 39.09 ± 4% 38.84 ± 5% ~ (p=0.947 n=30) 38.84 ± 3% ~ (p=0.854 n=30)
BuildABSL/page_cache.dirty/filesystem.bindfs-16 37.83 ± 3% 36.58 ± 4% ~ (p=0.057 n=30) 36.83 ± 5% ~ (p=0.314 n=30)
BuildABSL/page_cache.clean/filesystem.tmpfs-16 39.34 ± 3% 38.59 ± 4% ~ (p=0.350 n=30) 38.58 ± 4% ~ (p=0.300 n=30)
BuildABSL/page_cache.dirty/filesystem.tmpfs-16 37.83 ± 3% 36.08 ± 4% -4.64% (p=0.026 n=30) 36.58 ± 4% ~ (p=0.123 n=30)
BuildABSL/page_cache.clean/filesystem.rootfs-16 39.59 ± 4% 38.83 ± 3% ~ (p=0.485 n=30) 40.09 ± 5% ~ (p=0.971 n=30)
BuildABSL/page_cache.dirty/filesystem.rootfs-16 36.83 ± 3% 38.08 ± 5% ~ (p=0.307 n=30) 38.08 ± 1% ~ (p=0.242 n=30)
BuildABSL/page_cache.clean/filesystem.fusefs-16 38.34 ± 3% 37.59 ± 5% ~ (p=0.752 n=30) 38.59 ± 3% ~ (p=0.982 n=30)
BuildABSL/page_cache.dirty/filesystem.fusefs-16 37.58 ± 4% 38.08 ± 5% ~ (p=0.708 n=30) 36.08 ± 6% ~ (p=0.127 n=30)
BuildGRPC/page_cache.clean/filesystem.bindfs-16 212.7 ± 2% 211.0 ± 1% ~ (p=0.138 n=30) 211.2 ± 1% ~ (p=0.458 n=30)
BuildGRPC/page_cache.dirty/filesystem.bindfs-16 210.0 ± 1% 210.0 ± 1% ~ (p=0.542 n=30) 209.7 ± 1% ~ (p=0.665 n=30)
BuildGRPC/page_cache.clean/filesystem.rootfs-16 210.5 ± 1% 210.0 ± 1% ~ (p=0.423 n=30) 210.0 ± 1% ~ (p=0.142 n=30)
BuildGRPC/page_cache.dirty/filesystem.rootfs-16 210.2 ± 1% 209.0 ± 1% ~ (p=0.219 n=30) 209.5 ± 1% ~ (p=0.230 n=30)
geomean 67.62 66.97 -0.96% 67.12 -0.74%
```
The KVM platform benefits significantly from reduced nested page faults due to
huge pages, and to a lesser extent due to recycling:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
│ baseline │ expt │ expt2 │
│ sec/op │ sec/op vs base │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-12 43.11 ± 2% 39.35 ± 3% -8.71% (p=0.000 n=20) 38.10 ± 4% -11.63% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.bindfs-12 42.35 ± 3% 39.09 ± 4% -7.69% (p=0.000 n=20+19) 39.09 ± 5% -7.69% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.tmpfs-12 42.35 ± 3% 38.34 ± 5% -9.46% (p=0.000 n=20) 38.59 ± 3% -8.87% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.tmpfs-12 42.09 ± 1% 37.59 ± 4% -10.70% (p=0.000 n=20) 38.09 ± 4% -9.51% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.rootfs-12 42.85 ± 3% 38.84 ± 3% -9.35% (p=0.000 n=20) 39.09 ± 3% -8.77% (p=0.000 n=20+17)
BuildABSL/page_cache.dirty/filesystem.rootfs-12 41.85 ± 2% 39.59 ± 6% -5.40% (p=0.000 n=20+19) 38.09 ± 3% -9.00% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.fusefs-12 42.60 ± 2% 38.34 ± 2% -10.00% (p=0.000 n=20) 39.59 ± 3% -7.06% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.fusefs-12 42.09 ± 4% 39.09 ± 3% -7.13% (p=0.000 n=20) 38.09 ± 3% -9.52% (p=0.000 n=20+19)
BuildGRPC/page_cache.clean/filesystem.bindfs-12 207.7 ± 1% 206.4 ± 0% -0.60% (p=0.018 n=20) 205.9 ± 1% -0.85% (p=0.001 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.bindfs-12 206.9 ± 1% 206.9 ± 1% ~ (p=0.121 n=20) 204.4 ± 1% -1.22% (p=0.004 n=20+19)
BuildGRPC/page_cache.clean/filesystem.rootfs-12 207.7 ± 1% 204.9 ± 1% -1.33% (p=0.004 n=20) 203.9 ± 0% -1.81% (p=0.000 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.rootfs-12 206.9 ± 1% 204.9 ± 0% -0.97% (p=0.004 n=20+19) 203.9 ± 0% -1.45% (p=0.000 n=20+19)
geomean 71.97 67.63 -6.03% 67.28 -6.52%
```
PiperOrigin-RevId: 647771821
Either TotalHostMem or TotalMem are good candidates for limits
because in case either of these is set we should not be going over
them.
The motivations of this is to help catch syscalls causing allocations
with size values that are blatantly bad.
PiperOrigin-RevId: 633961720
This option, enabled via `runsc checkpoint --exclude-committed-zero-pages`,
instructs `pgalloc.MemoryFile.SaveTo()` to also exclude definitely-committed
zero pages from checkpointing (in addition to possibly-committed zero pages,
which are always scanned for and excluded). This is useful when the application
being checkpointed is known to have a large number of committed zero pages:
pages that (1) have been touched by application memory accesses, a syscall such
as read(), or page pinning by e.g. a driver, and (2) have not been subsequently
released by the application to the operating system by e.g. munmap() or
madvise(MADV_DONTNEED) (+ page unpinning if necessary), and (3) are filled with
zero bytes.
Minor changes:
- In `MemoryFile.updateUsageLocked()`, pass file offset to `checkCommitted` so
that `MemoryFile.SaveTo()`'s `checkCommitted` can use `FALLOC_FL_PUNCH_HOLE`
to decommit pages rather than `MADV_REMOVE` (which translates addresses to
file offsets and then invokes `FALLOC_FL_PUNCH_HOLE`).
- In `MemoryFile.SaveTo()`, buffer up to a hugepage worth of pages to decommit
rather than decommitting one page per syscall.
- Increment `MemoryFile.usageExpected` in `MemoryFile.LoadFrom()`, such that
the first following call to `MemoryFile.UpdateUsage()` might skip the call
to `MemoryFile.updateUsageLocked()` (if memory usage hasn't changed since
loading).
PiperOrigin-RevId: 632370455
MapInternal() returns a coherent memory mapping of the host file descriptor
represented by a memmap.File, in the sentry's address space. This is
principally used when the sentry needs to access the contents of application
memory (for e.g. syscall arguments passed by pointer, or the source/destination
of a write()/read() syscall); it usually looks up the memmap.Files backing
application addresses and obtains mappings via MapInternal().
/dev/nvidia-uvm cannot generally be mapped into the sentry's address space, for
reasons described by
https://github.com/google/gvisor/blob/master/g3doc/proposals/nvidia_driver_proxy.md#unified-virtual-memory-uvm
(in short, nvidia-uvm requires that a given page at file offset X can only be
mapped at address X). To allow the sentry to access the contents of such
mappings, make it possible for memmap.File.MapInternal() to indicate that a
fallback to buffered I/O is required, add interface methods
memmap.File.Buffer{Read,Write}At() to perform this buffered I/O, and implement
this fallback in the MM I/O path.
This CL does not use the new buffered I/O fallback anywhere; a following CL
adds it to nvproxy's nvidia-uvm.
Updates #10331
PiperOrigin-RevId: 629830825
Earlier we were always restoring pma.file to mm.mfp.MemoryFile(). However,
534c2e3ce0 ("Add support for saving PMAs referencing tmpfs filestore files.")
added support for saving PMAs that reference "private" pgalloc.MemoryFiles that
are different from mm.mfp.MemoryFile().
We achieve the correct restore by:
- Adding a "RestoreID" field to pgalloc.MemoryFile. Private MemoryFiles set
this with a vfs.RestoreID.String(). Non-private MemoryFile does not set it.
- MemoryFile struct is not savable by itself, but pma.file field is saved as a
string. We store the RestoreID string there.
- On restore, if RestoreID is "", then restore using CtxMemoryFile. If it has a
non-empty RestoreID, then restore using CtxMemoryFileMap.
- Cleanup: vfs.CtxFilesystemMemoryFileMap was moved to pgalloc.CtxMemoryFileMap
so we can now provide a pgalloc.MemoryFileMapFromContext() method which
cleans up some code. Also the key to this map (MemoryFileOpts.RestoreID)
belongs to pgalloc, so it seems like the right place to have this context.
PiperOrigin-RevId: 614903073
The safemem.Reader interface receiver was causing the implementation to escape
to heap (as at compile time, the compiler can not prove anything about the
implementation).
Using generics for io.ReadFullToBlocks() does not help in avoiding the heap
allocation.
With this change, we avoid at least these allocations:
- rw variable in kcov.Kcov.TaskWork().
- safemem.BlockSeqReader struct in in mm.MemoryManager.getPMAsInternalLocked().
- gr variable in fsutil.FileRangeSet.Fill().
- h variable in gofer.dentry.Translate().
- h variable in gofer.dentryReadWriter.ReadToBlocks().
Some of these are on hot paths for IO workloads.
PiperOrigin-RevId: 609805848
The rootfs is such a mount (with default overlay2=root:self configuration).
It should be possible to checkpoint this mount when application has active
VMAs that have allocated PMAs from the backing filestore file.
1479bc389e ("Add checkpoint/restore support for tmpfs with file backend.")
added support for saving such a tmpfs filestore file during checkpoint and
restoring it correctly. Hence, after restore, the VMAs and PMAs should be
valid.
PiperOrigin-RevId: 605766485
- Replace Add with TryInsertRange; for symmetry with RemoveRange, to establish
the convention that *Range methods perform an implicit search in the set, and
so that we can fork InsertRange which has Insert-like semantics (panics on
conflict), which the majority of callers want.
- Rename MergeRange and MergeAdjacent to MergeInsideRange and MergeOutsideRange
respectively; for the same convention, and to more clearly describe the
difference between these functions.
- Add MergePrev and MergeNext. These solve the longstanding problem of
requiring a separate call to Merge{Inside,Outside}Range (which will perform
additional searches) after mutating a set in a relatively simple manner.
- Add SplitBefore and SplitAfter, which are halves of Isolate. These are
slightly preferable to Isolate in many use cases for the latter (when
iterating segments within a range, only the first segment can include a key
before the start, so this saves some useless comparisons in almost every
iteration of such loops), and are useful in some more complex algorithms.
Also add LowerBoundSegmentSplitBefore and UpperBoundSegmentSplitAfter as
ergonomic aids for the former use case.
- Add {Visit,Mutate}[Full]Range, which are convenience wrappers around the
iterator API (including new functions) for simple use cases (and hence also
serve to demonstrate how the new iterator functions are used).
MutateFullRange in particular replaces ApplyContiguous and adds merging
during iteration.
- Add RemoveFullRange, which (analogous to {Visit,Mutate}FullRange) is a
variant of RemoveRange that checks that the range is fully covered by
segments.
- Add Unisolate, which combines MergePrev and MergeNext in the same way that
Isolate combines SplitBefore and SplitAfter. This is useful for merging after
mutation of a single segment.
- Add {First,Last,LowerBound,UpperBound}LargeEnoughGap, which are convenient
loop starters when using gap tracking.
- Replace SegmentDataSlices with FlatSegment, which is easier to use when
specifying "set literals" (as in tests).
- Make {prev,next}LargeEnoughGapHelper iterative rather than tail-recursive.
- Slightly optimize Iterator.{Prev,Next}NonEmpty: GapIterator.{Start,End} needs
to find the corresponding Iterator, so call Iterator.{Prev,Next}Segment
directly rather than doing so twice.
PiperOrigin-RevId: 583506148
MM.privateRefs counted pgalloc.MemoryFile references before platform.File
refcounting was added in cl/191858388. Since that CL, MM.privateRefs is
vestigial.
PiperOrigin-RevId: 581379213
It is an idea of running codespell as part of our presubmit checks.
Before enabling it for new changes, let's fix what it has found.
Signed-off-by: Andrei Vagin <[email protected]>
The total(sandbox) memory usage using the GetContainerMemoryUsage API will
return incorrect usage when called before calling the API for each individual
containers in the sandbox. This is because the memory usage for the containers
cgroup is not updated while calculating the total usage. This CL fixes it by
updating the usage for every child cgroup, which will return the correct memory
usage for the parent cgroup.
PiperOrigin-RevId: 574300913
Update the memmap IncRef method to pass memory cgroup id and store it in the
FrameRefSet which will be used for memory accounting. During DecRef, the
memCgID from the FrameRefSet will be retrieved and passed to MemoryLocked.Dec
to remove the memory from the cgroup.
PiperOrigin-RevId: 549656411
- Adds a new field in the usageInfo to store the memory cgroup id.
- Creates a map of cgroup ids and memory stats to track the memory per cgroup
in MemoryLocked struct.
- Introduces new methods to increment, decrement, move, copy and get the total
memory usage per cgroup.
PiperOrigin-RevId: 549148091
Earlier we had another method for AllocateAndPopulate. This change moves the
additional (optional) parameters to pgalloc.AllocOpts.
* Makes `Reader` parameter optional. Not all callers have data to fill into
the allocated pages. Avoids having to define no-op readers.
* Avoid calling MapInternal() when `Reader == nil` and
`Mode != AllocateAndWritePopulate`.
* Avoid calling safemem.ReadFullToBlocks() when `Reader == nil`.
* Move AllocationMode upgrade/downgrade logic to callsites. This makes
Allocate() simpler.
PiperOrigin-RevId: 547688619
For memory accounting per task, the memory cgroup id of the task is required.
This CL retrieves the cgroup id from the task context and stores it in the
AllocOpts struct which will later be used to account for memory usage.
PiperOrigin-RevId: 546996195
When an application makes a huge write, it is better to pre-fault all the pages
that will later be touched by runtime.memmove() via safemem.CopySeq(). Earlier
we were calling MemoryFile.Allocate(), which does not populate the pages.
Otherwise, these pages fault one at a time as they are touched, which is really
slow due to context switching.
Note that this optimization only works for shmem-backed MemoryFiles. This means
that writes to unallocated regions of tmpfs mounts (like /tmp) will be faster.
rootfs (overlay) does not benefit because even though it has an upper tmpfs
layer, the MemoryFile used is disk-backed.
We only prepopulate if more than 1 page is being written to. This is because
the syscall path is susceptible to certain overheads that the page fault path
does not have. For example seccomp-bpf and syscall auditing.
The improved large writes in tmpfs is evidenced by (on Linux >=5.14):
- Before
```
$ docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 3.10212 s, 1.3 GB/s
```
- After
```
docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 2.38347 s, 1.7 GB/s
```
In this workload, this change reduces latency by 23% (3.10212s -> 2.38347s).
PiperOrigin-RevId: 544056038
Pre-populating pages on fallocate(2) when the MemoryFile is backed by a disk
is wasteful.
It is currently wasteful because the mlock+munlock mechanism used in
pgalloc.tryPopulate() to populate pages does not work with some disk-backed
filesystems like ext4. We end up faulting read-only pages, which need to
be re-faulted on write anyways.
Later we will update pgalloc.tryPopulate() to use madvice(MADV_POPULATE_WRITE).
It wouldn't work in that case too because even though it does fault writable
pages, they are written back to disk before the application can write to them.
And so they will be re-faulted on write. This would actually worsen the
performance because now we have the extra cost of writeback.
Hence, avoid populating pages for disk-backed MemoryFile on fallocate(2).
Instead we just commit the pages to satisfy fallocate(2) semantics.
When the application writes to these file offsets, the pages will fault
normally.
To allow this distinction between "committing" and "populating" pages in
addition to allocating them, the pgalloc API has been extended by introducing
`pgalloc.AllocationMode` enum. It allows the caller to select if pages should
be only-allocated, allocated+committed or allocated+committed+populated.
Here are the benchmarking results:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ sec/op │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-4 100.73 ± 12% 99.72 ± 58% ~ (p=0.805 n=7)
BuildGRPC/page_cache.clean/filesystem.bindfs-4 703.5 ± 1% 690.9 ± 1% -1.78% (p=0.004 n=7)
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 2.008m ± 0% 2.009m ± 25% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=0.318 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 2.510m ± 0% 2.509m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 2.009m ± 0% 2.009m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.097 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 3.010m ± 17% 2.510m ± 0% ~ (p=0.053 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 2.009m ± 25% 2.008m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.259 n=7)
RubySpecTest/page_cache.clean/filesystem.bindfs-4 63.15 ± 2% 61.65 ± 2% -2.38% (p=0.011 n=7)
geomean 34.15m 33.48m -1.94%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ bandwidth.bytes_per_second │ bandwidth.bytes_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 939.6M ± 2% 944.7M ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 2.937G ± 5% 2.873G ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 3.139G ± 7% 3.112G ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 1.355G ± 3% 1.382G ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 3.913G ± 10% 3.785G ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.692G ± 11% 3.855G ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 788.4M ± 2% 628.6M ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 1346.1M ± 5% 940.4M ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.646G ± 5% 1.173G ± 3% -28.75% (p=0.001 n=7)
geomean 1.882G 1.699G -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ io_ops.ops_per_second │ io_ops.ops_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 229.4k ± 2% 230.6k ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 44.82k ± 5% 43.84k ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 2.994k ± 7% 2.967k ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 330.7k ± 3% 337.3k ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 59.70k ± 10% 57.76k ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.521k ± 11% 3.676k ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 192.5k ± 2% 153.5k ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 20.54k ± 5% 14.35k ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.570k ± 5% 1.119k ± 3% -28.73% (p=0.001 n=7)
geomean 28.72k 25.93k -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ load.sec │ load.sec vs base │
RubySpecTest/page_cache.clean/filesystem.bindfs-4 9.430 ± 7% 9.150 ± 7% -2.97% (p=0.026 n=7)
```
The broader filesystem benchmarks (ABSL, gRPC, Ruby) all improved with this
change. Improvement is ~2-3% reduction in absolute time to run these
workloads. However, the FIO microbenchmark is a little more complicated.
The latency metric shows improvement (for blockSize=4K & filesystem=rootfs).
It is stable for other blockSize values. But FIO bandwidth seems to have
regressed. We think this is an accounting gimmick. We suspect that sec/op
includes the app fallocate() and bandwidth does not. Since this change moves
ext4_mpage_readpages() from app fallocate() to the page faults that occur
during app write(), bandwidth appears to have regressed.
Given that the broader filesystem benchmarks look good and that this is more
consistent with Linux (app fallocate() now only commits disk space rather than
also materializing memory pages), this change should be an improvement.
PiperOrigin-RevId: 542331863
Earlier we were using the same filestore for all containers in the
sandbox. This was based on the assumption that all containers in a
k8s pod have the same runsc configuration. We get rid of our
dependence on that assumption. Subcontainers can independently use
overlay configurations.
This is needed because in k8s, different ephemeral storage limits can
be enforced on containers in the same pod. This behavior is required
to get limit enforcement to work correctly on a per container basis.
This change also moves the ownership of overlay filestore to tmpfs.
Now that each container has its own overlay-backing MemoryFile, it will
be much harder to manage lifecycle of these MemoryFiles as containers
are created and destroyed.
Instead, move ownership to tmpfs mount. When the overlay upper layer tmpfs
is unmounted, it will destroy the MemoryFile.
We also extract out IMA work around logic from NewMemoryFile() and perform
it outside the sandbox. This is done because now tmpfs is creating the
MemoryFile. By then the sandbox is already running with seccomp filters
installed that do not allow certain mmap(2) calls that the said work around
uses. We hit the same issue for subcontainers. When they are started,
the sandbox is already running with seccomp filters. Added a new
MemoryFileOpts field for this which defaults to old behavior.
PiperOrigin-RevId: 505892085
MemoryFile.findReclaimable() immediately exits when MemoryFile.Destroy() is
called, even if MemoryFile.reclaim set is not empty. If the MemoryFile is
backed by a host filesystem based file, then just closing MemoryFile.FD() does
not release resources held by the MemoryFile. To release the disk space held by
the host file, it needs to be decommitted. Otherwise, we can leave around large
host files.
Guard this change in behavior behind a flag in MemoryFileOpts which defaults to
the old behavior of not decommitting the host file.
PiperOrigin-RevId: 505884271
--overlay2 flag supersedes --overlay flag. It allows more granular
configuration for overlayfs in runsc. It does so in two ways:
1. Allows to apply overlay on all mounts or only the root mount.
--overlay applies overlay to all mounts.
2. Allows to specify if overlay's upper layer should be backed by
container memory or disk. --overlay always used container memory.
Allowing tmpfs to be backed by a file on disk prevents the container
memory from bloating up. Note that the tmpfs filesystem tree will
still be stored in sentry memory.
Using overlay on the root filesystem, helps avoid expensive
communication with the gofer process. The root filesystem of the
container is not preserved across container lifecycle. So we don't
need to keep updating the host filesystem, which will anyways be
destroyed once the container is destroyed. It is wasted effort.
Instead we keep all the changes to the root filesystem in tmpfs which
is directly accessible by the sentry.
The host file is created as an unnamed file using O_TMPFILE. Support
has been added for sub-containers too. Save/restore support is
still lacking.
Co-authored-by: Andrei Vagin <[email protected]>
PiperOrigin-RevId: 491988485
MemoryFile allocated offsets from the top down. This can generate
fragmentation when allocations happen in the opposite direction.
Given that applications control the order of allocations, use simple
heuristics to track the last faulted address to determine whether memory
file should be allocated from the top-down or bottom-up.
Here is the number of PMAs used across all processes in some common
workloads:
Workload | Before | After | Diff
---------|---------|---------|------
rustc | 135,269 | 1,090 | 0.8%
mysql | 16,466 | 4,576 | 28%
nginx | 1,738 | 1,609 | 93%
jenkins | 13,830 | 6,380 | 46%
redis | 334 | 308 | 92%
absl | 297,419 | 292,121 | 98%
Here is the direction of PMA allocations:
Workload | BottomUp | TopDown |
---------|-----------|---------|
mysql | 13,849 | 1,990 |
nginx | 783 | 628 |
jenkins | 13,922 | 2,333 |
redis | 88 | 109 |
absl | 212,403 | 44,019 |
Tests were done on Intel x64.
PiperOrigin-RevId: 409014690
This change makes the checklocks analyzer considerable more powerful, adding:
* The ability to traverse complex structures, e.g. to have multiple nested
fields as part of the annotation.
* The ability to resolve simple anonymous functions and closures, and perform
lock analysis across these invocations. This does not apply to closures that
are passed elsewhere, since it is not possible to know the context in which
they might be invoked.
* The ability to annotate return values in addition to receivers and other
parameters, with the same complex structures noted above.
* Ignoring locking semantics for "fresh" objects, i.e. objects that are
allocated in the local frame (typically a new-style function).
* Sanity checking of locking state across block transitions and returns, to
ensure that no unexpected locks are held.
Note that initially, most of these findings are excluded by a comprehensive
nogo.yaml. The findings that are included are fundamental lock violations.
The changes here should be relatively low risk, minor refactorings to either
include necessary annotations to simplify the code structure (in general
removing closures in favor of methods) so that the analyzer can be easily
track the lock state.
This change additional includes two changes to nogo itself:
* Sanity checking of all types to ensure that the binary and ast-derived
types have a consistent objectpath, to prevent the bug above from occurring
silently (and causing much confusion). This also requires a trick in
order to ensure that serialized facts are consumable downstream. This can
be removed with https://go-review.googlesource.com/c/tools/+/331789 merged.
* A minor refactoring to isolation the objdump settings in its own package.
This was originally used to implement the sanity check above, but this
information is now being passed another way. The minor refactor is preserved
however, since it cleans up the code slightly and is minimal risk.
PiperOrigin-RevId: 382613300
Update all instances of the above errors to the faster linuxerr implementation.
With the temporary linuxerr.Equals(), no logical changes are made.
PiperOrigin-RevId: 382306655
Split usermem package to help remove syserror dependency in go_marshal.
New hostarch package contains code not dependent on syserror.
PiperOrigin-RevId: 365651233
This validates that struct fields if annotated with "// checklocks:mu" where
"mu" is a mutex field in the same struct then access to the field is only
done with "mu" locked.
All types that are guarded by a mutex must be annotated with
// +checklocks:<mutex field name>
For more details please refer to README.md.
PiperOrigin-RevId: 360729328
The syscall package has been deprecated in favor of golang.org/x/sys.
Note that syscall is still used in the following places:
- pkg/sentry/socket/hostinet/stack.go: some netlink related functionalities
are not yet available in golang.org/x/sys.
- syscall.Stat_t is still used in some places because os.FileInfo.Sys() still
returns it and not unix.Stat_t.
Updates #214
PiperOrigin-RevId: 360701387
This change makes the following changes:
- Unlocks MemoryFile.mu while calling mincore (checkCommitted) because mincore
can take a really long time. Accordingly looks up the segment in the tree
tree again and handles changes to the segment.
- MemoryFile.UpdateUsage() can now only be called at frequency at most 100Hz.
100 Hz = linux.CLOCKS_PER_SEC.
Co-authored-by: Jamie Liu <[email protected]>
PiperOrigin-RevId: 337865250
Ayush Ranjan