Syzkaller came up with workloads that fallocate(2) 1 TB in /tmp. The host
mlock(2) or madvise(2) syscalls on memfd(2) files end up hanging for multiple
minutes in such situations causing the watchdog to mark the calling goroutine
as stuck. memfd(2) files have not size limits.
Linux fails such fallocate(2) attempts in /tmp with ENOSPC.
In Linux tmpfs (shmem), when size= mount option is not specified, the default
size limit for the mount is set to 50% of physical RAM size. But in gVisor, it
is set to MaxInt64. Which is why Linux fails with ENOSPC and gVisor doesn't.
In runsc, the physcial RAM size is already exposed to the containerized
application via `/proc/meminfo` which uses `usage.*TotalMemoryBytes`. These
fields are configured using the `MemTotal:` field from host `/proc/meminfo`.
So use that information to set the default size limit correctly.
Reported-by: [email protected]
PiperOrigin-RevId: 548252095
Prepopulating pages for disk-backed MemoryFiles has proved to be futile.
The mf.MapInternal()+safemem.CopySeq() approach used right now incurs a lot of
page faults without page population. Page-by-page faults incurs a lot of
context switching.
On the other hand, the write syscall makes one context switch to kernel, and
faults all the pages that are touched during write. Note that safemem.CopySeq()
avoids a syscall and hence can be faster sometimes when the underlying page
is populated. But with disk writebacks, it is hard to predict/account what is
populated. Writebacks can happen asynchronously based on system load.
Benchmark results show that FIO write performance improves a lot on rootfs:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ benchout.runsc-before │ benchout.runsc-after │
│ sec/op │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-4 90.20 ± 2% 89.44 ± 1% ~ (p=0.382 n=8)
BuildGRPC/page_cache.clean/filesystem.bindfs-4 626.0 ± 1% 626.8 ± 0% ~ (p=0.505 n=8)
RubySpecTest/page_cache.clean/filesystem.bindfs-4 52.11 ± 1% 52.37 ± 1% ~ (p=0.105 n=8)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 2.509m ± 0% 2.509m ± 0% ~ (p=0.878 n=8)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 2.009m ± 0% 1.507m ± 0% -24.98% (p=0.000 n=8)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 2.008m ± 0% 1.508m ± 0% -24.90% (p=0.000 n=8)
│ benchout.runsc-before │ benchout.runsc-after │
│ bandwidth.bytes_per_second │ bandwidth.bytes_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 649.1M ± 2% 705.2M ± 2% +8.64% (p=0.000 n=8)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 991.1M ± 1% 1499.1M ± 3% +51.25% (p=0.000 n=8)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.198G ± 2% 1.945G ± 2% +62.34% (p=0.000 n=8)
│ benchout.runsc-before │ benchout.runsc-after │
│ io_ops.ops_per_second │ io_ops.ops_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 158.5k ± 2% 172.2k ± 2% +8.64% (p=0.000 n=8)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 15.12k ± 1% 22.87k ± 3% +51.25% (p=0.000 n=8)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.143k ± 2% 1.855k ± 2% +62.34% (p=0.000 n=8)
│ benchout.runsc-before │ benchout.runsc-after │
│ load.sec │ load.sec vs base │
RubySpecTest/page_cache.clean/filesystem.bindfs-4 7.555 ± 1% 7.585 ± 1% ~ (p=0.457 n=8)
```
PiperOrigin-RevId: 548021076
Earlier we had another method for AllocateAndPopulate. This change moves the
additional (optional) parameters to pgalloc.AllocOpts.
* Makes `Reader` parameter optional. Not all callers have data to fill into
the allocated pages. Avoids having to define no-op readers.
* Avoid calling MapInternal() when `Reader == nil` and
`Mode != AllocateAndWritePopulate`.
* Avoid calling safemem.ReadFullToBlocks() when `Reader == nil`.
* Move AllocationMode upgrade/downgrade logic to callsites. This makes
Allocate() simpler.
PiperOrigin-RevId: 547688619
For memory accounting per task, the memory cgroup id of the task is required.
This CL retrieves the cgroup id from the task context and stores it in the
AllocOpts struct which will later be used to account for memory usage.
PiperOrigin-RevId: 546996195
When an application makes a huge write, it is better to pre-fault all the pages
that will later be touched by runtime.memmove() via safemem.CopySeq(). Earlier
we were calling MemoryFile.Allocate(), which does not populate the pages.
Otherwise, these pages fault one at a time as they are touched, which is really
slow due to context switching.
Note that this optimization only works for shmem-backed MemoryFiles. This means
that writes to unallocated regions of tmpfs mounts (like /tmp) will be faster.
rootfs (overlay) does not benefit because even though it has an upper tmpfs
layer, the MemoryFile used is disk-backed.
We only prepopulate if more than 1 page is being written to. This is because
the syscall path is susceptible to certain overheads that the page fault path
does not have. For example seccomp-bpf and syscall auditing.
The improved large writes in tmpfs is evidenced by (on Linux >=5.14):
- Before
```
$ docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 3.10212 s, 1.3 GB/s
```
- After
```
docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 2.38347 s, 1.7 GB/s
```
In this workload, this change reduces latency by 23% (3.10212s -> 2.38347s).
PiperOrigin-RevId: 544056038
Pre-populating pages on fallocate(2) when the MemoryFile is backed by a disk
is wasteful.
It is currently wasteful because the mlock+munlock mechanism used in
pgalloc.tryPopulate() to populate pages does not work with some disk-backed
filesystems like ext4. We end up faulting read-only pages, which need to
be re-faulted on write anyways.
Later we will update pgalloc.tryPopulate() to use madvice(MADV_POPULATE_WRITE).
It wouldn't work in that case too because even though it does fault writable
pages, they are written back to disk before the application can write to them.
And so they will be re-faulted on write. This would actually worsen the
performance because now we have the extra cost of writeback.
Hence, avoid populating pages for disk-backed MemoryFile on fallocate(2).
Instead we just commit the pages to satisfy fallocate(2) semantics.
When the application writes to these file offsets, the pages will fault
normally.
To allow this distinction between "committing" and "populating" pages in
addition to allocating them, the pgalloc API has been extended by introducing
`pgalloc.AllocationMode` enum. It allows the caller to select if pages should
be only-allocated, allocated+committed or allocated+committed+populated.
Here are the benchmarking results:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ sec/op │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-4 100.73 ± 12% 99.72 ± 58% ~ (p=0.805 n=7)
BuildGRPC/page_cache.clean/filesystem.bindfs-4 703.5 ± 1% 690.9 ± 1% -1.78% (p=0.004 n=7)
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 2.008m ± 0% 2.009m ± 25% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=0.318 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 2.510m ± 0% 2.509m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 2.009m ± 0% 2.009m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.097 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 3.010m ± 17% 2.510m ± 0% ~ (p=0.053 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 2.009m ± 25% 2.008m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.259 n=7)
RubySpecTest/page_cache.clean/filesystem.bindfs-4 63.15 ± 2% 61.65 ± 2% -2.38% (p=0.011 n=7)
geomean 34.15m 33.48m -1.94%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ bandwidth.bytes_per_second │ bandwidth.bytes_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 939.6M ± 2% 944.7M ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 2.937G ± 5% 2.873G ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 3.139G ± 7% 3.112G ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 1.355G ± 3% 1.382G ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 3.913G ± 10% 3.785G ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.692G ± 11% 3.855G ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 788.4M ± 2% 628.6M ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 1346.1M ± 5% 940.4M ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.646G ± 5% 1.173G ± 3% -28.75% (p=0.001 n=7)
geomean 1.882G 1.699G -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ io_ops.ops_per_second │ io_ops.ops_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 229.4k ± 2% 230.6k ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 44.82k ± 5% 43.84k ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 2.994k ± 7% 2.967k ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 330.7k ± 3% 337.3k ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 59.70k ± 10% 57.76k ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.521k ± 11% 3.676k ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 192.5k ± 2% 153.5k ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 20.54k ± 5% 14.35k ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.570k ± 5% 1.119k ± 3% -28.73% (p=0.001 n=7)
geomean 28.72k 25.93k -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ load.sec │ load.sec vs base │
RubySpecTest/page_cache.clean/filesystem.bindfs-4 9.430 ± 7% 9.150 ± 7% -2.97% (p=0.026 n=7)
```
The broader filesystem benchmarks (ABSL, gRPC, Ruby) all improved with this
change. Improvement is ~2-3% reduction in absolute time to run these
workloads. However, the FIO microbenchmark is a little more complicated.
The latency metric shows improvement (for blockSize=4K & filesystem=rootfs).
It is stable for other blockSize values. But FIO bandwidth seems to have
regressed. We think this is an accounting gimmick. We suspect that sec/op
includes the app fallocate() and bandwidth does not. Since this change moves
ext4_mpage_readpages() from app fallocate() to the page faults that occur
during app write(), bandwidth appears to have regressed.
Given that the broader filesystem benchmarks look good and that this is more
consistent with Linux (app fallocate() now only commits disk space rather than
also materializing memory pages), this change should be an improvement.
PiperOrigin-RevId: 542331863
We plan on making --overlay2=root:self the default for runsc. That will be a
risky change which might need rollbacks. This change is in preparation for
that. We manually set --overlay2=none in places where we don't want the
overlay configuration to impact. This change should be a noop. The intention
for this change is to make the risky change very small and limited to just
flipping a flag.
PiperOrigin-RevId: 513930702
Also add the missing HugePage equivalents of Page functions.
The fix for golang/go#56280, which first appears in Go 1.20, is needed to
prevent this CL from regressing performance:
Before fix:
```
TEXT gvisor/pkg/sentry/mm/mm.(*MemoryManager).SetNumaPolicy(SB) gvisor/pkg/sentry/mm/syscalls.go
...
addr.go:75 0x7f000054e35e 488d053bc2b100 LEAQ gvisor/pkg/hostarch/hostarch..dict.IsPageAligned[gvisor/pkg/hostarch/hostarch.Addr](SB), AX
addr.go:75 0x7f000054e365 e8961cc4ff CALL gvisor/pkg/hostarch/hostarch.IsPageAligned[go.shape.uintptr](SB)
```
After fix:
```
TEXT gvisor/pkg/sentry/mm/mm.(*MemoryManager).SetNumaPolicy(SB) gvisor/pkg/sentry/mm/syscalls.go
...
addr.go:75 0x7f00004da61a 90 NOPL
addr.go:75 0x7f00004da61b 0f1f440000 NOPL 0(AX)(AX*1)
sizes_util.go:57 0x7f00004da620 48f7c3ff0f0000 TESTQ $0xfff, BX
syscalls.go:1021 0x7f00004da627 0f855f010000 JNE 0x7f00004da78c
```
PiperOrigin-RevId: 507608080
Earlier we were using the same filestore for all containers in the
sandbox. This was based on the assumption that all containers in a
k8s pod have the same runsc configuration. We get rid of our
dependence on that assumption. Subcontainers can independently use
overlay configurations.
This is needed because in k8s, different ephemeral storage limits can
be enforced on containers in the same pod. This behavior is required
to get limit enforcement to work correctly on a per container basis.
This change also moves the ownership of overlay filestore to tmpfs.
Now that each container has its own overlay-backing MemoryFile, it will
be much harder to manage lifecycle of these MemoryFiles as containers
are created and destroyed.
Instead, move ownership to tmpfs mount. When the overlay upper layer tmpfs
is unmounted, it will destroy the MemoryFile.
We also extract out IMA work around logic from NewMemoryFile() and perform
it outside the sandbox. This is done because now tmpfs is creating the
MemoryFile. By then the sandbox is already running with seccomp filters
installed that do not allow certain mmap(2) calls that the said work around
uses. We hit the same issue for subcontainers. When they are started,
the sandbox is already running with seccomp filters. Added a new
MemoryFileOpts field for this which defaults to old behavior.
PiperOrigin-RevId: 505892085
All filesystems except tmpfs had the same bug regarding handling symlinks
in the terminal path component. When they tried to open(2) such a symlink
file, these implementations called into stepLocked() with
Then these implementations also called vfs.HandleSymlink() which would
also advance rp (when rp.Done() was already true).
This was tripping the invariants checks in sentry/vfs/debug.go.
PiperOrigin-RevId: 504098263
FileRangeSet.Fill() can fill in gaps partially and fail. Earlier, we were
over-correcting on failure by unaccounting everything that was reserved for the
operation. But that can lead to accounting errors in case of partial Fill()
failures. In such a case, when the file is later deleted, we would again
unaccount all the pages that were allocated via the partial Fill(), which will
causes accounting to go negative and sentry will panic.
Also did some refactoring for clearer code.
PiperOrigin-RevId: 493935801
We were already returning this large size via statfs(2). But there was no
accounting being done against it. Even after page allocations, there was no
change in values returned by statfs(2) about number of pages free.
PHP runtime test ext/standard/tests/file/disk_free_space_basic.phpt and Python
runtime test test_shutil tests for this. This change is required to get these
tests to pass with overlayfs.
PiperOrigin-RevId: 493903408
This is likely to be a speed-up in the common case when contention between
threads is low. In such situations, generally atomic operations are faster.
PiperOrigin-RevId: 493731992
--overlay2 flag supersedes --overlay flag. It allows more granular
configuration for overlayfs in runsc. It does so in two ways:
1. Allows to apply overlay on all mounts or only the root mount.
--overlay applies overlay to all mounts.
2. Allows to specify if overlay's upper layer should be backed by
container memory or disk. --overlay always used container memory.
Allowing tmpfs to be backed by a file on disk prevents the container
memory from bloating up. Note that the tmpfs filesystem tree will
still be stored in sentry memory.
Using overlay on the root filesystem, helps avoid expensive
communication with the gofer process. The root filesystem of the
container is not preserved across container lifecycle. So we don't
need to keep updating the host filesystem, which will anyways be
destroyed once the container is destroyed. It is wasted effort.
Instead we keep all the changes to the root filesystem in tmpfs which
is directly accessible by the sentry.
The host file is created as an unnamed file using O_TMPFILE. Support
has been added for sub-containers too. Save/restore support is
still lacking.
Co-authored-by: Andrei Vagin <[email protected]>
PiperOrigin-RevId: 491988485
Implementing memory allocation taking into consideration the fact
that the page allocation does not changes whenever the size of a
file changes. For instance, ftruncate may increase the size of a
file without actually allocating that size. So it changes when the
pages are actually allocated.
Also making amends to allow for partial allocations. For instance,
a write, mmap can allocate pages as long as the size option
is not exhausted.
Fixes#7827
PiperOrigin-RevId: 470426017
This is consistent with Linux. Earlier, the allocate() handler in tmpfs was
simply calling into truncate(). However fallocate(2) must also reserve the disk
space for the range specified by offset and length. This is because, after a
successful fallocate(2) call, writes in that range must not fail due to lack of
disk space.
Updates #7827
PiperOrigin-RevId: 465203457
In Linux, the callers of fs/namei.c:do_renameat2() do this check
by calling fs/namei.c:getname().
This is important of consistency with Linux. Also, without this
we can see panics in overlayfs rename if new file's name is too
long. Because upperlayer RenameAt operation would succeed but
SetXattr operation will fail leading to an inconsistent state.
PiperOrigin-RevId: 464173793
`vfs.FileDescription` in general handles all inotify work. Only
SetStat, SetXattr, RemoveXattr and Iterdirents require FDImpls to
handle the Inotify work. This leads to repeated code. And for
kernfs users, there are too many FD implementors. So it becomes
difficult plumbing Inotify consistenly through all FDImpls.
So make it consistent that all Inotify work is handled by
`vfs.FileDescription`, not FDImpls.
PiperOrigin-RevId: 461751122
Adding test case to check empty size parsing in Linux.
If a value is not passed with size the mount(2) should return EINVAL.
Handling empty size parsing in gVisor.
PiperOrigin-RevId: 448132149
- Removed old TODO from VFS1. We will not support this option on VFS1.
- Enhanced comments about how tmpfs pages are accounted for by write(2)s.
PiperOrigin-RevId: 446072121
Fixing partial write memory allocation.
Accounting for memory allocation in hard links.
Adding test cases to check for hard link memory allocation for the same.
PiperOrigin-RevId: 443742013
When tmpfs is mounted with `size` option it imposes a limit for
the tmpfs mount. The total size of all files in that mount must not exceed this
limit. See tmpfs(5) for more details. When this option was not supported and
specifying it caused mount(2) to return EINVAL instead of ENOSPC.
In Linux, tmpfs charges symlinks, regular files and directories against this
`size` limit. All accounting is done on the granularity of page size. The `size`
limit and all individual file sizes are rounded up to number of pages while
performing calculations. gVisor has aimed to replicate the same behavior.
One difference from Linux is that if the `size` option is not specified, then
Linux uses 50% of physical RAM as the `size`. We don't replicate this behavior
in gVisor as the actual size of host physical RAM should not be exposed to
application. In gVisor, if `size` option is not specified, then no limit is
imposed. This is consistent with the behavior before this change.
Added syscall tests for regular files and symlink.
PiperOrigin-RevId: 442686814
All atomic 64 bit ints are changed to atomicbitops.(Ui|I)nt64. A nogo checker
enforces that sync/atomic 64 bit functions are not called.
For reviewers: the interesting changes are in the atomicbitops and checkaligned
packages.
Why do this?
- It is very easy to accidentally use atomic values without sync/atomic funcs.
- We have checkatomics, but this is optional and is forgotten in several places.
- Using a type+checker to enforce this seems less error prone and simpler.
- We get NoCopy protection.
- Use of 64 bit atomics can break 32 bit builds. We have types to handle this
without any runtime cost, so we might as well use them.
PiperOrigin-RevId: 440473398
Since refsvfs2 are used in netstack, we should use atomicbitops to avoid
breaking 32-bit builds.
On 64-bit builds there is no performance difference.
PiperOrigin-RevId: 439687980
Linux exempts special files (sockets, named pipes, and character/block device
special files) from mount writability checks in both open
(fs/open.c:do_dentry_open()) and access (fs/open.c:do_faccessat()) paths. We
don't currently do so in open (in VFS1 or VFS2) or access (in VFS1), so we
don't do so in VFS2 access either.
PiperOrigin-RevId: 439398002
This change adapts the existing context to use more suitable non-channel-based
methods. This is a requisite for migrating the kernel internals to a
sleeper-based notification mechanism.
The last uses of amutex outside those migrated as part of this change were
dropped in a previous change. Since amutex depends on the channel-based
implementation, this package is also deleted as part of this change.
PiperOrigin-RevId: 415189675
These amutex lock uses are limited to vfs1 and provide questionable utility.
They protect only offset access, and not blocking operations. In order to
completely remove amutex, drop these uses. The amutex package will be removed
in a subsequent commit, which migrates other (less questionable) uses to a new
Context API.
PiperOrigin-RevId: 409716979
MemoryFile allocated offsets from the top down. This can generate
fragmentation when allocations happen in the opposite direction.
Given that applications control the order of allocations, use simple
heuristics to track the last faulted address to determine whether memory
file should be allocated from the top-down or bottom-up.
Here is the number of PMAs used across all processes in some common
workloads:
Workload | Before | After | Diff
---------|---------|---------|------
rustc | 135,269 | 1,090 | 0.8%
mysql | 16,466 | 4,576 | 28%
nginx | 1,738 | 1,609 | 93%
jenkins | 13,830 | 6,380 | 46%
redis | 334 | 308 | 92%
absl | 297,419 | 292,121 | 98%
Here is the direction of PMA allocations:
Workload | BottomUp | TopDown |
---------|-----------|---------|
mysql | 13,849 | 1,990 |
nginx | 783 | 628 |
jenkins | 13,922 | 2,333 |
redis | 88 | 109 |
absl | 212,403 | 44,019 |
Tests were done on Intel x64.
PiperOrigin-RevId: 409014690
