While there is a difference between flavors of this ioctl (`TCSETS`,
`TCSETSF`, `TCSETSW`), in practice it seems not to matter. We should let
users call `TCSETSF` and can implement those differences if necessary.
This issue was reported by Syzkaller. The stacktrace shows
forgetDeadMountpoint(..., skipDecRef=true) being called. But unlockMounts()
is still shown DecRef()-ing the objects. PopDelayedDecRefs() should be clearing
vfs.toDecRef.
Also make gofer.dentry.evictLocked() DecRef the returned objects. It was
ignoring the return value earlier.
Reported-by: [email protected]
PiperOrigin-RevId: 560164788
- Adds methods to Enter, Leave and Migrate the tasks for memory controller.
- Update the memory cgroup id when the task enters and leaves the cgroups.
PiperOrigin-RevId: 555568875
DynamicBytesFD is used for cgroup and proc files. In Linux, these file systems
don't set splice_write callbacks.
Reported-by: [email protected]
PiperOrigin-RevId: 553818208
This change introduces the nsfs file system. Each new namespace allocates
a new nsfs inode.
Here are reasons why we need these inodes:
* each namespace has to have an unique id.
* proc/pid/ns/ contains one entry for each namespace. Bind mounting one of
the files in this directory to somewhere else in the filesystem keeps the
corresponding namespace alive even if all processes currently in
the namespace terminate.
* setns() allows the calling process to join an existing namespace specified
by a file descriptor.
PiperOrigin-RevId: 550694515
The TPU userspace driver needs access to specific PCI device information
located in Linux sysfs. We mirror the sysfs paths the driver reads on the host
in the Sentry sysfs. This way we can ensure we only expose the host device
information that's strictly necessary for TPU to run.
PiperOrigin-RevId: 550005271
Update the memmap IncRef method to pass memory cgroup id and store it in the
FrameRefSet which will be used for memory accounting. During DecRef, the
memCgID from the FrameRefSet will be retrieved and passed to MemoryLocked.Dec
to remove the memory from the cgroup.
PiperOrigin-RevId: 549656411
Syzkaller came up with workloads that fallocate(2) 1 TB in /tmp. The host
mlock(2) or madvise(2) syscalls on memfd(2) files end up hanging for multiple
minutes in such situations causing the watchdog to mark the calling goroutine
as stuck. memfd(2) files have not size limits.
Linux fails such fallocate(2) attempts in /tmp with ENOSPC.
In Linux tmpfs (shmem), when size= mount option is not specified, the default
size limit for the mount is set to 50% of physical RAM size. But in gVisor, it
is set to MaxInt64. Which is why Linux fails with ENOSPC and gVisor doesn't.
In runsc, the physcial RAM size is already exposed to the containerized
application via `/proc/meminfo` which uses `usage.*TotalMemoryBytes`. These
fields are configured using the `MemTotal:` field from host `/proc/meminfo`.
So use that information to set the default size limit correctly.
Reported-by: [email protected]
PiperOrigin-RevId: 548252095
The flag is only enforced in lisafs gofer as of now.
This change plumbs a gofer client flag which disallowing opening FIFO from the
host filesystem.
PiperOrigin-RevId: 548193558
Prepopulating pages for disk-backed MemoryFiles has proved to be futile.
The mf.MapInternal()+safemem.CopySeq() approach used right now incurs a lot of
page faults without page population. Page-by-page faults incurs a lot of
context switching.
On the other hand, the write syscall makes one context switch to kernel, and
faults all the pages that are touched during write. Note that safemem.CopySeq()
avoids a syscall and hence can be faster sometimes when the underlying page
is populated. But with disk writebacks, it is hard to predict/account what is
populated. Writebacks can happen asynchronously based on system load.
Benchmark results show that FIO write performance improves a lot on rootfs:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ benchout.runsc-before │ benchout.runsc-after │
│ sec/op │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-4 90.20 ± 2% 89.44 ± 1% ~ (p=0.382 n=8)
BuildGRPC/page_cache.clean/filesystem.bindfs-4 626.0 ± 1% 626.8 ± 0% ~ (p=0.505 n=8)
RubySpecTest/page_cache.clean/filesystem.bindfs-4 52.11 ± 1% 52.37 ± 1% ~ (p=0.105 n=8)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 2.509m ± 0% 2.509m ± 0% ~ (p=0.878 n=8)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 2.009m ± 0% 1.507m ± 0% -24.98% (p=0.000 n=8)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 2.008m ± 0% 1.508m ± 0% -24.90% (p=0.000 n=8)
│ benchout.runsc-before │ benchout.runsc-after │
│ bandwidth.bytes_per_second │ bandwidth.bytes_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 649.1M ± 2% 705.2M ± 2% +8.64% (p=0.000 n=8)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 991.1M ± 1% 1499.1M ± 3% +51.25% (p=0.000 n=8)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.198G ± 2% 1.945G ± 2% +62.34% (p=0.000 n=8)
│ benchout.runsc-before │ benchout.runsc-after │
│ io_ops.ops_per_second │ io_ops.ops_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 158.5k ± 2% 172.2k ± 2% +8.64% (p=0.000 n=8)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 15.12k ± 1% 22.87k ± 3% +51.25% (p=0.000 n=8)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.143k ± 2% 1.855k ± 2% +62.34% (p=0.000 n=8)
│ benchout.runsc-before │ benchout.runsc-after │
│ load.sec │ load.sec vs base │
RubySpecTest/page_cache.clean/filesystem.bindfs-4 7.555 ± 1% 7.585 ± 1% ~ (p=0.457 n=8)
```
PiperOrigin-RevId: 548021076
Earlier we had another method for AllocateAndPopulate. This change moves the
additional (optional) parameters to pgalloc.AllocOpts.
* Makes `Reader` parameter optional. Not all callers have data to fill into
the allocated pages. Avoids having to define no-op readers.
* Avoid calling MapInternal() when `Reader == nil` and
`Mode != AllocateAndWritePopulate`.
* Avoid calling safemem.ReadFullToBlocks() when `Reader == nil`.
* Move AllocationMode upgrade/downgrade logic to callsites. This makes
Allocate() simpler.
PiperOrigin-RevId: 547688619
For memory accounting per task, the memory cgroup id of the task is required.
This CL retrieves the cgroup id from the task context and stores it in the
AllocOpts struct which will later be used to account for memory usage.
PiperOrigin-RevId: 546996195
In Linux, oom_adj_write treats an input buffer as a string. It means that
it parses only the part before the first zero byte.
PiperOrigin-RevId: 544711418
When an application makes a huge write, it is better to pre-fault all the pages
that will later be touched by runtime.memmove() via safemem.CopySeq(). Earlier
we were calling MemoryFile.Allocate(), which does not populate the pages.
Otherwise, these pages fault one at a time as they are touched, which is really
slow due to context switching.
Note that this optimization only works for shmem-backed MemoryFiles. This means
that writes to unallocated regions of tmpfs mounts (like /tmp) will be faster.
rootfs (overlay) does not benefit because even though it has an upper tmpfs
layer, the MemoryFile used is disk-backed.
We only prepopulate if more than 1 page is being written to. This is because
the syscall path is susceptible to certain overheads that the page fault path
does not have. For example seccomp-bpf and syscall auditing.
The improved large writes in tmpfs is evidenced by (on Linux >=5.14):
- Before
```
$ docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 3.10212 s, 1.3 GB/s
```
- After
```
docker run --runtime=runsc --rm ubuntu bash -c "dd if=/dev/zero of=/tmp/file.txt bs=40960 count=100000"
100000+0 records in
100000+0 records out
4096000000 bytes (4.1 GB, 3.8 GiB) copied, 2.38347 s, 1.7 GB/s
```
In this workload, this change reduces latency by 23% (3.10212s -> 2.38347s).
PiperOrigin-RevId: 544056038
Pre-populating pages on fallocate(2) when the MemoryFile is backed by a disk
is wasteful.
It is currently wasteful because the mlock+munlock mechanism used in
pgalloc.tryPopulate() to populate pages does not work with some disk-backed
filesystems like ext4. We end up faulting read-only pages, which need to
be re-faulted on write anyways.
Later we will update pgalloc.tryPopulate() to use madvice(MADV_POPULATE_WRITE).
It wouldn't work in that case too because even though it does fault writable
pages, they are written back to disk before the application can write to them.
And so they will be re-faulted on write. This would actually worsen the
performance because now we have the extra cost of writeback.
Hence, avoid populating pages for disk-backed MemoryFile on fallocate(2).
Instead we just commit the pages to satisfy fallocate(2) semantics.
When the application writes to these file offsets, the pages will fault
normally.
To allow this distinction between "committing" and "populating" pages in
addition to allocating them, the pgalloc API has been extended by introducing
`pgalloc.AllocationMode` enum. It allows the caller to select if pages should
be only-allocated, allocated+committed or allocated+committed+populated.
Here are the benchmarking results:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ sec/op │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-4 100.73 ± 12% 99.72 ± 58% ~ (p=0.805 n=7)
BuildGRPC/page_cache.clean/filesystem.bindfs-4 703.5 ± 1% 690.9 ± 1% -1.78% (p=0.004 n=7)
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 2.008m ± 0% 2.009m ± 25% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=0.318 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 1.507m ± 0% 1.507m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 2.510m ± 0% 2.509m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 2.009m ± 0% 2.009m ± 0% ~ (p=0.805 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.097 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 3.010m ± 17% 2.510m ± 0% ~ (p=0.053 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 2.009m ± 25% 2.008m ± 0% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 2.009m ± 0% 2.008m ± 0% ~ (p=0.259 n=7)
RubySpecTest/page_cache.clean/filesystem.bindfs-4 63.15 ± 2% 61.65 ± 2% -2.38% (p=0.011 n=7)
geomean 34.15m 33.48m -1.94%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ bandwidth.bytes_per_second │ bandwidth.bytes_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 939.6M ± 2% 944.7M ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 2.937G ± 5% 2.873G ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 3.139G ± 7% 3.112G ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 1.355G ± 3% 1.382G ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 3.913G ± 10% 3.785G ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.692G ± 11% 3.855G ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 788.4M ± 2% 628.6M ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 1346.1M ± 5% 940.4M ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.646G ± 5% 1.173G ± 3% -28.75% (p=0.001 n=7)
geomean 1.882G 1.699G -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ io_ops.ops_per_second │ io_ops.ops_per_second vs base │
Fio/operation.write/blockSize.4K/filesystem.bindfs-4 229.4k ± 2% 230.6k ± 4% ~ (p=0.737 n=7)
Fio/operation.write/blockSize.64K/filesystem.bindfs-4 44.82k ± 5% 43.84k ± 2% ~ (p=0.216 n=7)
Fio/operation.write/blockSize.1024K/filesystem.bindfs-4 2.994k ± 7% 2.967k ± 6% ~ (p=0.879 n=7)
Fio/operation.write/blockSize.4K/filesystem.tmpfs-4 330.7k ± 3% 337.3k ± 5% ~ (p=0.090 n=7)
Fio/operation.write/blockSize.64K/filesystem.tmpfs-4 59.70k ± 10% 57.76k ± 10% ~ (p=0.779 n=7)
Fio/operation.write/blockSize.1024K/filesystem.tmpfs-4 3.521k ± 11% 3.676k ± 6% ~ (p=1.000 n=7)
Fio/operation.write/blockSize.4K/filesystem.rootfs-4 192.5k ± 2% 153.5k ± 1% -20.26% (p=0.001 n=7)
Fio/operation.write/blockSize.64K/filesystem.rootfs-4 20.54k ± 5% 14.35k ± 6% -30.13% (p=0.001 n=7)
Fio/operation.write/blockSize.1024K/filesystem.rootfs-4 1.570k ± 5% 1.119k ± 3% -28.73% (p=0.001 n=7)
geomean 28.72k 25.93k -9.72%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-cl1 │
│ load.sec │ load.sec vs base │
RubySpecTest/page_cache.clean/filesystem.bindfs-4 9.430 ± 7% 9.150 ± 7% -2.97% (p=0.026 n=7)
```
The broader filesystem benchmarks (ABSL, gRPC, Ruby) all improved with this
change. Improvement is ~2-3% reduction in absolute time to run these
workloads. However, the FIO microbenchmark is a little more complicated.
The latency metric shows improvement (for blockSize=4K & filesystem=rootfs).
It is stable for other blockSize values. But FIO bandwidth seems to have
regressed. We think this is an accounting gimmick. We suspect that sec/op
includes the app fallocate() and bandwidth does not. Since this change moves
ext4_mpage_readpages() from app fallocate() to the page faults that occur
during app write(), bandwidth appears to have regressed.
Given that the broader filesystem benchmarks look good and that this is more
consistent with Linux (app fallocate() now only commits disk space rather than
also materializing memory pages), this change should be an improvement.
PiperOrigin-RevId: 542331863
DecRef'ing dead mountpoints while holding a filesystem lock can cause
nested locking, since we need to lock the filesystem to access the cache.
Reported-by: [email protected]
PiperOrigin-RevId: 539834013
Readonly mounts or readonly path mappings will return EROFS or EPERM even if
the file exists at that location. So relying on MkdirAt() to return EEXIST can
yield incorrect results.
Instead call MkdirAllAt(), which uses StatAt() to check for existence. This
yields accurate results.
PiperOrigin-RevId: 539228542
This is an optimization that will prevent needless lookups on lower layers for
directories. This is analogous to the Linux commit 97c684cc9110 ("ovl: create
directories inside merged parent opaque").
If mkdir(2) does succeed, then none of the lower layers have a file at that
position, otherwise mkdir(2) would have failed with EEXIST.
Changes on the lower layer underneath a new upper layer directory are not
visible. See the following for example:
```
$ sudo mount -t overlay -o lowerdir=lower,upperdir=upper,workdir=workdir none overlay
$ mkdir -p overlay/dir/dir1
$ mkdir -p lower/dir/dir2
$ ls overlay/dir/
dir1
```
Benchmarking shows that this change cuts ABSL build time by 9.5%!
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-opt │
│ sec/op │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-8 71.16 ± 12% 64.40 ± 3% -9.50% (p=0.003 n=8)
BuildGRPC/page_cache.clean/filesystem.bindfs-8 418.2 ± 1% 419.9 ± 1% ~ (p=0.721 n=8)
RubySpecTest/page_cache.clean/filesystem.bindfs-8 76.67 ± 5% 75.17 ± 1% ~ (p=0.382 n=8)
geomean 131.6 126.7 -3.78%
│ /tmp/benchout.runsc │ /tmp/benchout.runsc-opt │
│ load.sec │ load.sec vs base │
RubySpecTest/page_cache.clean/filesystem.bindfs-8 12.95 ± 6% 12.80 ± 5% ~ (p=0.625 n=8)
```
PiperOrigin-RevId: 538823897
These inodes can never be part of a filesystem tree. They are nameless and
never have a parent.
This allows us to avoid taking a lock in kernfs.InotifyWithParent for such
anonymous inodes.
PiperOrigin-RevId: 538823227
