The idea behind conditionally compiled metrics originally was to use them in
hotpaths for profiling purposes. This CL makes that possible by outputting
declared metrics in TSV format, which can be used to track custom events at
runtime in relatively high resolution.
Usage:
1. Optionally enable compilation of runsc conditionally-compiled metrics
by passing in condmetric_profiling to the Go tags.
2. Add these flags to runsc:
- [Required] --profiling-metrics-log=/tmp/some.csv
- [Optional] --profiling-metrics=/task/syscalls,/task/faults
- If this flag is not specified it will monitor all
conditionally-compiled metrics by default.
- [Optional] --profiling-metrics-rate-us=10000
Some future improvements:
- Flag to output a metric-difference between timestamps instead of
constant accumulation.
- Output a gnuplot command along with the data.
- Current monitoring resolution is limited by what time.Sleep allows.
This can be overcome by spinning/yielding when lower monitoring
rates are requested.
PiperOrigin-RevId: 560849611
Set(String()) should be an idempotent operation. This is a useful property
which allows us to generate args while re-execing the same process. Setting
`--flag-name=val.String()` should work.
PiperOrigin-RevId: 552598313
Users/callers of config.Overlay2 relied on its internal functioning/layout.
The stable contract is at the --overlay2 flag level. So changed callers to use
that contract instead via Overlay2.Set().
Unexported fields to encapsulate the type correctly.
PiperOrigin-RevId: 530818635
This allows `runsc` subcommands that don't start sandboxes to run in
restricted contexts without printing a warning about not being able to set
`RLIMIT_MEMLOCK` when the ability to do so this doesn't matter.
In particular, this helps with `runsc metric-server`, which can be locked down
to run with very little capabilities.
A previous version of this change had moved this to the beginning of the
`runsc boot` subcommand code. However, this doesn't work, because `runsc boot`
runs as an unprivileged user (`nobody`) and does not have `CAP_SYS_RESOURCE`.
Prior to that change, all `runsc` invocations tried to call `setrlimit`, so
what happened in practice is that `runsc create` (running as `root`) would
call `setrlimit`, and then `runsc boot` would inherit the `RLIM_INFINITY` and
would therefore never actually call `setrlimit` by itself. When moving the
`setrlimit` code to only run within `runsc boot`, suddenly the `runsc boot`
invocation found itself in a context where it started trying to call
`setrlimit`, which would silently fail.
This approach has the downside of having the side-effect of needlessly setting
`RLIM_INFINITY` on the calling `runsc` process. This was effectively what was
already happening prior to moving this code into `runsc boot` anyway, so this
should be OK. The alternative would be to add yet another intermediate
subcommand before `runsc boot` which runs with `CAP_SYS_RESOURCE`, then calls
`setrlimit`, then drops `CAP_SYS_RESOURCE`, then execs `runsc boot`, but that
seems like adding a lot more extra complexity to the boot process than is
warranted for this feature.
Thanks to Ayush Ranjan for bisecting the performance regression down to this
change.
Ran benchmarks and performance is comparable to before moving `setrlimit` code
within `runsc boot`.
PiperOrigin-RevId: 521916084
Add portforward comand so that we can use runsc to forward connections
to container ports. This will eventually be supported in k8s.
PiperOrigin-RevId: 520739913
This is helpful so that it can be imported form other packages without import
loops.
This will be used in a follow-up change to add the version string as a
per-sandbox metric metadata label.
PiperOrigin-RevId: 519002695
directfs is a feature much like host networking. The gofer security boundary is
dropped for additional performance. The sentry is allowed to directly access the
host filesystem to access or mutate files in order to service application
syscalls. This mode bypasses the gofer completely.
With directfs, the sentry is run in the root user namespace. (Similar to how
hostinet works.) The sentry also runs with additional capabilities that allow
it to bypass file permissions checks, change file owner, chroot, etc. Seccomp
filters are loosened around the sentry to allow it to make filesystem syscalls
directly to the host.
The sandbox still runs with an empty pivot root. The gofer donates FDs for the
root mount and all bind mounts so that the sentry can perform filesystem
operations using them.
There is a dummy gofer process in directfs mode. The gofer sets up all bind
mounts and donates host FDs of the mount point to the sandbox. The gofer dies
with the sandbox. All gofer mount options are supported with directfs. directfs
mounts are set up similarly to how gofer mounts are set up.
PiperOrigin-RevId: 513634936
This flag is added for tests that need to trigger a panic in the sentry
kernel. Only done for x86_64 which does have a dedicated syscall number for
afs_syscall; ARM does not.
PiperOrigin-RevId: 512731631
This allows other `runsc` subcommands to run in restricted contexts without
printing a warning about not being able to set `RLIMIT_MEMLOCK` in contexts
where this doesn't matter.
In particular, this helps with `runsc metric-server`, which can be locked down
to run with very little capabilities.
PiperOrigin-RevId: 508491474
This is to make it clear that the host file will be created inside this
directory. This also makes it look cleaner when other medium options are added
later.
PiperOrigin-RevId: 505033408
The metrics server implements the following interface:
- GET `/metrics`: Serves Prometheus metrics.
- POST `/runsc-metrics`: Contains administrative endpoints.
All of them require the `root` argument to be specified, and match the one
that the server expects. This is used to avoid confusing multiple instances
of the `runsc` metrics server.
- POST `/runsc-metrics/healthcheck`: Used by clients to verify that the
server is running and is the expected metric server.
This change has no tests, but coverage is provided in a later change that
provides an end-to-end container tests that the metric server works and
exports data faithfully.
This change is part of a series of changes to support Prometheus-style metrics
in `runsc`. Doing so requires making several seemingly-odd design decisions,
due to the following architectural constraints:
- Prometheus requires an HTTP server serving the `/metrics` endpoint.
- For performance reasons, the `runsc boot` process cannot run the `netpoller`
goroutine.
- Since we don't want to write our own HTTP server implementation, this
means the HTTP endpoint has to be served by a separate process that
remains running during the lifetime of the container.
- The `runsc boot` process is untrusted.
- This means we cannot trust metrics data that comes out of the Sentry.
Therefore, there needs to be an elaborate dance where we pre-register
metric metadata before starting any untrusted workload. Then, the server
relaying the metric data must verify the validity of metric values against
this metric metadata. This avoids leaking metrics, cardinality blow-ups,
and other such DoS vectors.
- This feature needs to be easy-to-use in a typical Docker setting.
- This means having the ability to just say
`--metrics-server=localhost:1337` in the `runsc` runtime entry in
`/etc/docker/daemon.json` and have that Just Work(TM), even when multiple
containers are running.
- Since only one process may listen on a port at a given time, this means
the metric server needs to be able to multiplex requests out to multiple
running sandboxes, and remain alive for the entire duration of either of
these sandboxes.
- For this reason, the metrics server runs *outside* of the usual
per-container cgroups.
- This also saves system resources by not running one server per sandbox.
- The metrics server must be exposed to the outside world, and cannot assume
that its clients are trustworthy.
- For this reason, a metrics server is bound to a runtime root directory,
and double-checks all that the sandboxes it is asked to follow actually
exist in this root directory.
PiperOrigin-RevId: 503254345
The current io_uring support is very limited and experimental. Disable
it by default, and add a flag to enable it for testing.
PiperOrigin-RevId: 500760451
This subcommand prints a sandbox's instrumentation data in Prometheus format
to stdout.
This change is part of a series of changes to support Prometheus-style metrics
in `runsc`. Doing so requires making several seemingly-odd design decisions,
due to the following architectural constraints:
- Prometheus requires an HTTP server serving the `/metrics` endpoint.
- For performance reasons, the `runsc boot` process cannot run the `netpoller`
goroutine.
- Since we don't want to write our own HTTP server implementation, this
means the HTTP endpoint has to be served by a separate process that
remains running during the lifetime of the container.
- The `runsc boot` process is untrusted.
- This means we cannot trust metrics data that comes out of the Sentry.
Therefore, there needs to be an elaborate dance where we pre-register
metric metadata before starting any untrusted workload. Then, the server
relaying the metric data must verify the validity of metric values against
this metric metadata. This avoids leaking metrics, cardinality blow-ups,
and other such DoS vectors.
- This feature needs to be easy-to-use in a typical Docker setting.
- This means having the ability to just say
`--metrics-server=localhost:1337` in the `runsc` runtime entry in
`/etc/docker/daemon.json` and have that Just Work(TM), even when multiple
containers are running.
- Since only one process may listen on a port at a given time, this means
the metric server needs to be able to multiplex requests out to multiple
running sandboxes, and remain alive for the entire duration of either of
these sandboxes. However, it should also die when there are no sandboxes,
so that we don't end up with leftover metric servers lying around.
- For this reason, the metrics server runs *outside* of the usual
per-container cgroups.
- This also saves system resources by not running one server per sandbox.
- The metrics server must be exposed to the outside world, and cannot assume
that its clients are trustworthy.
- For this reason, a metrics server is bound to a runtime root directory,
and double-checks all that the sandboxes it is asked to follow actually
exist in this root directory.
PiperOrigin-RevId: 498067941
--overlay2 flag supersedes --overlay flag. It allows more granular
configuration for overlayfs in runsc. It does so in two ways:
1. Allows to apply overlay on all mounts or only the root mount.
--overlay applies overlay to all mounts.
2. Allows to specify if overlay's upper layer should be backed by
container memory or disk. --overlay always used container memory.
Allowing tmpfs to be backed by a file on disk prevents the container
memory from bloating up. Note that the tmpfs filesystem tree will
still be stored in sentry memory.
Using overlay on the root filesystem, helps avoid expensive
communication with the gofer process. The root filesystem of the
container is not preserved across container lifecycle. So we don't
need to keep updating the host filesystem, which will anyways be
destroyed once the container is destroyed. It is wasted effort.
Instead we keep all the changes to the root filesystem in tmpfs which
is directly accessible by the sentry.
The host file is created as an unnamed file using O_TMPFILE. Support
has been added for sub-containers too. Save/restore support is
still lacking.
Co-authored-by: Andrei Vagin <[email protected]>
PiperOrigin-RevId: 491988485
The sandbox process is started with the procfs mount and then it is umounted
before executing guest processes.
Here are a few reasons why we need /proc in the sandbox namespace:
* /proc/self/exe is used to re-executed itself without capabilities and
with a specific uid,gid, etc. execveat() can be protected by some security
policies. For example, on my workstation, it fails tomoyo_bprm_check_security
checks.
* runsc reads /proc/cpuinfo, /proc/sys/vm/mmap_min_addr, /proc/self/auxv,
/proc/sys/kernel/cap_last_cap.
* The kvm platform reads /proc/self/maps to get current mappings.
PiperOrigin-RevId: 488733897
Also make `runsc debug` write to stdout and log file to make it
easier to see the output when running the command manually.
PiperOrigin-RevId: 470042053
The trace commands allows a user to manipulate trace sessions.
`runsc trace create <name> --config <file>` => creates a new trace session
`runsc trace delete <name>` => deletes an existing trace session
`runsc trace list` => lists all running trace sessions
`runsc trace metadata` => lists all point with their respective optional and
context fields
This allows trace sessions to be created/deleted on a running sandbox. Note
that the system currently only allows a single trace session to exist, named
'Default'. Attempts to manipulate other sessions will error out.
Updates #4805
PiperOrigin-RevId: 447815153
First, this change moves the internal config API to use flag.FlagSet, which
allows more flexibility and fixes many test usages.
Second, the runtime flags are validated during install. The platform is opened
and a warning issued if this fails, but this is not fatal.
This change requires moving the Makefile to --test_env, since the attribute
is not properly supported by test targets. Therefore, the targets can use args
while the Makefile must pass in configuration via --test_env.
PiperOrigin-RevId: 428048274
In some cases, it may be desirable to prebuild binaries and run all tests,
for example to run benchmarks with various experiments. Allow the top-level
Makefile to support this by checking for a STAGED_BINARIES variable.
PiperOrigin-RevId: 410673120
lisafs is only supported in VFS2. Added a runsc flag which enables lisafs.
When the flag is enabled, the gofer process and the client communicate using
lisafs protocol instead of 9P.
Added a filesystem option in fsimpl/gofer which indicates if lisafs is being
used. That will be used to gate lisafs on the gofer client.
Note that this change does not make the gofer client use lisafs just yet.
Updates #5465
PiperOrigin-RevId: 397917844
In the previous spot, there was a roughly 50% chance that leak checking would
actually run. Move it to the waitContainer() call on the root container, where
it is guaranteed to run before the sandbox process is terminated. Add it to
runsc/cli/main.go as well for good measure, in case the sandbox exit path does
not involve waitContainer().
PiperOrigin-RevId: 369329796
Add a coverage-report flag that will cause the sandbox to generate a coverage
report (with suffix .cov) in the debug log directory upon exiting. For the
report to be generated, runsc must have been built with the following Bazel
flags: `--collect_code_coverage --instrumentation_filter=...`.
With coverage reports, we should be able to aggregate results across all tests
to surface code coverage statistics for the project as a whole.
The report is simply a text file with each line representing a covered block
as `file:start_line.start_col,end_line.end_col`. Note that this is similar to
the format of coverage reports generated with `go test -coverprofile`,
although we omit the count and number of statements, which are not useful for
us.
Some simple ways of getting coverage reports:
bazel test <some_test> --collect_code_coverage \
--instrumentation_filter=//pkg/...
bazel build //runsc --collect_code_coverage \
--instrumentation_filter=//pkg/...
runsc -coverage-report=dir/ <other_flags> do ...
PiperOrigin-RevId: 368952911
Implement a new runsc command to set up a sandbox with verityfs and
run the measure tool. This is loosely forked from the do command, and
currently requires the caller to provide the measure tool binary.
PiperOrigin-RevId: 366553769
The syscall package has been deprecated in favor of golang.org/x/sys.
Note that syscall is still used in some places because the following don't seem
to have an equivalent in unix package:
- syscall.SysProcIDMap
- syscall.Credential
Updates #214
PiperOrigin-RevId: 361381490
This command takes instruction pointers from stdin and converts them into their
corresponding file names and line/column numbers in the runsc source code. The
inputs are not interpreted as actual addresses, but as synthetic values that are
exposed through /sys/kernel/debug/kcov. One can extract coverage information
from kcov and translate those values into locations in the source code by
running symbolize on the same runsc binary.
This will allow us to generate syzkaller coverage reports.
PiperOrigin-RevId: 347089624
A few images were broken with respect to aarch64. We should now
be able to run push-all-images with ARCH=aarch64 as part of the
regular continuous integration builds, and add aarch64 smoke tests
(via user emulation for now) to the regular test suite (future).
PiperOrigin-RevId: 346685462
Using the newer bazel rules necessitates a transition from proto1 to
proto2. In order to resolve the incompatibility between proto2 and
gogoproto, the cri runtimeoptions proto must be vendored.
Further, some of the semantics of bazel caching changed during the
transition. It is now necessary to:
- Ensure that :gopath depends only on pure library targets, as the
propagation of go_binary build attributes (pure, static) will
affected the generated files (though content remains the same,
there are conflicts with respect to the gopath).
- Update bazel.mk to include the possibility of binaries in the
bazel-out directory, as it will now put runsc and others there.
This required some refinements to the mechanism of extracting
paths, since some the existing regex resulted in false positives.
- Change nogo rules to prevent escape generation on binary targets.
For some reason, the newer version of bazel attempted to run the
nogo analysis on the binary targets, which fails due to the fact
that objdump does not work on the final binary. This must be due
to a change in the semantics of aspects in bazel3.
PiperOrigin-RevId: 337958324
