`runsc boot` still works, so single-file installations continue to work
for now.
No statistically significant change in startup latency, but unlocks the
ability to do per-platform PGO.
There were several issues with how runsc log and profile files were generated:
- Variables like "%ID" in flag-specified file paths were substituted by the
runsc shim in `pkg/shim/v1/runsccmd/utils.go:replaceID()`. Consequently, runsc
commands that did *not* invoke the shim missed this logic. Notably, Docker
invokes runsc via `runc shim`, so Docker use cases were missing this
feature.
- Variable support across flags was inconsistent. For example, the --debug-log
flag supported variables like ID, TIMESTAMP, COMMAND, and TEST, while
--profile-cpu supported ID, TIMESTAMP, YYYY, MM, etc.
- "%ID%" was intended to be substituted with the sandbox ID, but the shim was
incorrectly substituting it with the container ID.
- Variables specified via flag annotations were not substituted because the
shim did not process flag annotations.
- `specutils.ReadSpec() => ReadSpecFromFile() => fixSpec()` had the side effect
of modifying the `Config` object itself with annotation-based flags, rather
than just fixing the spec.
- Flag annotations were applied to Config for only 6 runsc commands (via
`fixSpec()`): boot, gofer, create, restore, run, and start. Other commands
did not see flags set via annotations.
- When using flag annotations to generate debug logs, only boot and gofer logs
were generated. This is because the debug file was created by the runsc CLI,
but flag annotations were processed later by the sub-command implementation.
Only `boot` and `gofer` had the correct configuration because `runsc create`
had already fixed its config by the time it invoked them.
- The parent directories of the configured file paths is created via
os.MkdirAll() only for a subset of the flags. This is inconsistent.
This change fixes all above-mentioned issues:
- Moves all variable substitution logic from the shim to the runsc CLI.
- Centralizes the logic for opening and formatting paths for log and profile
files within `specutils`.
- Introduces `specutils.LogFileOpts` to standardize variable substitution in
log and profile paths.
- Refactors profile file handling to use `DonateProfileFDs`, leveraging
`LogFileOpts` and `DonateLogFile` for consistent path substitution.
- Extracts `specutils.FixConfig()` from `specutils.fixSpec()` to only call it
when necessary. This prevents side effects; `fixSpec()` now only fixes the OCI
spec, and `FixConfig()` only fixes `config.Config`. The config is now fixed
uniformly for all commands via the runsc CLI.
- Standarizes the variables supported across all file path flags:
- `%TIMESTAMP%`: Replaced with the start-up time of the command with format
`<yyyymmdd-hhmmss.uuuuuu>`.
- `%COMMAND%`: Replaced with the command name.
- `%TEST%`: Replaced with the value of the environment variable named
`-TESTONLY-test-name-env`. The value is fetched from container spec.
- `%ID%`: Replaced with the sandbox ID.
- `%CID%`: Replaced with the container ID.
- Open all log files consistently with mode 0644 and create intermediate
directories with mode 0755.
To facilitate these improvements, the `subcommands.Command` interface has been
extended via `util.SubCommand` to include `FetchSpec(conf, flatSet)`.
- All runsc sub-commands must implement this interface.
- `FetchSpec` extracts the container ID from the command line arguments and
ensures the OCI spec is loaded (either from the bundle or container) for
configuration processing.
- Caching is added to all sub-commands to reuse the work done by `FetchSpec`
during `Execute()`.
Note: `runsc tar` and `runsc trace` implement `FetchSpec` as a no-op. These
commands have nested sub-commands that make spec pre-fetching complex. Variable
substitution is currently not supported for their file path flags.
PiperOrigin-RevId: 877758843
When using runsc exec with --detach but without --pid-file, the
command fails because the generated --pid-file flag is incorrectly
appended at the end of the argument list.
This patch fixes this issue by inserting it after the exec subcommand,
before the container ID.
Signed-off-by: Tianyu Zhou <[email protected]>
There is already some support for privileged executables before this change:
file capabilities were taken into account when creating new processes inside
the sandbox from external instigation (but not during execve(2)). set-user-id
and set-group-id bits were never taken into account, and the Sentry behaved
as if NO_NEW_PRIVS is always true. After this change, execve(2) and the
external Exec entry point both take the set-user(group)-id bits and file
capabilities into account.
Because NO_NEW_PRIVS can no longer be assumed to be always set, a couple of
extra checks to prevent privilege elevation during execve(2) are needed:
the presence of a ptracer and the use of a shared fsContext. Moreover,
execve had to be serialized with seccomp(2) when it is syncing filters to
all threads in a group. Serializing execve with PTRACE_ATTACH and seccomp
required some Task runState gymnastics.
Task dumpability is reset when privilege elevation takes place and privilege
elevation is denied for binaries living on nosuid mounts. To fix an old bug
where some aux vectors were populated using data from the old pre-execve
creds, the computation of the (file-privilege-augmented) creds during execve
now takes place before the stack for the new program is set up.
gvisor users who thus far have exec'd into the sandbox with UID as (sandbox)
root will lose the caps an all-powerful namespace root'd have if: 1) the binary
is a suid/guid binary 2) the owner of the binary is host root and there is
no mapping for host root in the host user namespace of the sandbox. To avoid
this surprise, the suid/sgid bits are ignored by default. Users can opt-in
via the --allow-suid flag.
One result of this change is that sudo now works inside the sandbox.
PiperOrigin-RevId: 794414101
Runsc will not split any value for a given `env`, the
behavior is consistent with runc. Each env variable
will be passed by respective `env`.
Fix#11885
PiperOrigin-RevId: 780337902
Previously we were only looking at stdin, which could be a pty but other stdio
fds might be redirected. In that case, we can incorrectly end up using the
stdin fd as *the* console fd, and sending all stdout/stderr to that FD,
ignoring the redirect.
Note that the behavior was actually flaky because the mechanism for choosing
which stdio fd to treat as *the* pty fd is non-deterministic (due to the map
iteration in fdimport/fdimport.go:Import) and so sometimes we would choose
the correct one.
This CL also cleans up `argsFromProcess` and `argsFromCLI`, which were setting
their `FilePayload` unnecessarily, since it is always set in `Execute`.
Fixes#11350Fixes#11349
PiperOrigin-RevId: 716733446
This is consistent with runc. This fixes several bugs with runsc exec:
- When --process flag is specified, the process spec should be validated. The
process spec should not inherit values from the OCI spec except capabilities.
Earlier, we were setting WorkingDirectory and Envv from the spec if these
were not set in the process file.
- When --process flag is not specified, we should use the Process defined in
the container spec as the base and append the following flags onto that
process spec. Earlier if these flags were specified, we were not using the
container spec values and just setting to these passed flags, hence making it
look like runsc is "clearing" these fields when their flags are passed.
- additional-gids
- cap
- env
- When --process flag is not specified, we should use the following values
defined in the container spec's Process. Those values should be selectively
overridden when the corresponding flag is set. Earlier, we were always using
the flag values, even when the flag was not set. One implication was that we
were always running with UID=GID=0 when --process and --user are not set.
- user
- cwd
- When --cap is set, it should not append to the Inheritable capabilities
defined in the spec. And it should only be appended to Ambient if Inheritable
in the original spec is non-empty.
Fixes#11108
PiperOrigin-RevId: 694642077
Set(String()) should be an idempotent operation. This is a useful property
which allows us to generate args while re-execing the same process. Setting
`--flag-name=val.String()` should work.
PiperOrigin-RevId: 552598313
This commit adds support for program execution via a host file
descriptor. To make use of this feature, the host file descriptor must
be provided to the --exec-fd argument. For example,
exec 3</usr/bin/echo
runsc exec --exec-fd=3 mycontainer hello world
will run the host's echo binary inside gVisor. In this case, "hello" is
supplied to echo as argv[0]. As a result, the output of the above
command is "world".
This feature is useful for bootstrapping unknown guest environments and
allows static executables to perform setup actions inside the container
while they need not be part of the guest file system.
GitHub pull request #8634 concluded that commit 9f521bd should be
merged. However, the changes since commit feb40ac got lost in merge
1f331d1. This commit is the lost diff.
This commit implements file descriptor passing from the host to the
guest. It implements a --pass-fd flag that can be specified multiple
times with FD numbers from the host that will be inserted into the file
descriptor table of the guest.
When a terminal is used, signals are now forwared to the foreground
process inside the sandbox. As a result, control commands like Ctrl+C
can be handled by the terminal in the guest.
Set stdio ownership based on the container's user to ensure the
user can open/read/write to/from stdios.
1. stdios in the host are changed to have the owner be the same
uid/gid of the process running the sandbox. This ensures that the
sandbox has full control over it.
2. stdios owner owner inside the sandbox is changed to match the
container's user to give access inside the container and make it
behave the same as runc.
Fixes#6180
PiperOrigin-RevId: 384347009
The syscall package has been deprecated in favor of golang.org/x/sys.
Note that syscall is still used in some places because the following don't seem
to have an equivalent in unix package:
- syscall.SysProcIDMap
- syscall.Credential
Updates #214
PiperOrigin-RevId: 361381490
This allows to find all containers inside a sandbox more efficiently.
This operation is required every time a container starts and stops,
and previously required loading *all* container state files to check
whether the container belonged to the sandbox.
Apert from being inneficient, it has caused problems when state files
are stale or corrupt, causing inavalability to create any container.
Also adjust commands `list` and `debug` to skip over files that fail
to load.
Resolves#5052
PiperOrigin-RevId: 348050637
When OOM score adjustment needs to be set, all the containers need to be
loaded to find all containers that belong to the sandbox. However, each
load signals the container to ensure it is still alive. OOM score
adjustment is set during creation and deletion of every container, generating
a flood of signals to all containers. The fix removes the signal check
when it's not needed.
There is also a race fetching OOM score adjustment value from the parent when
the sandbox exits at the same time (the time it took to signal containers above
made this window quite large). The fix is to store the original value
in the sandbox state file and use it when the value needs to be restored.
Also add more logging and made the existing ones more consistent to help with
debugging.
PiperOrigin-RevId: 340940799
'docker exec' was getting CAP_NET_RAW even when --net-raw=false
because it was not filtered out from when copying container's
capabilities.
PiperOrigin-RevId: 272260451
Go was going to change the behavior of SysProcAttr.Ctty such that it must be an
FD in the *parent* FD table:
https://go-review.googlesource.com/c/go/+/178919/
However, after some debate, it was decided that this change was too
backwards-incompatible, and so it was reverted.
https://github.com/golang/go/issues/29458
The behavior going forward is unchanged: the Ctty FD must be an FD in the
*child* FD table.
PiperOrigin-RevId: 255228476
An upcoming change in Go 1.13 [1] changes the semantics of the SysProcAttr.Ctty
field. Prior to the change, the FD must be an FD in the child process's FD
table (aka "post-shuffle"). After the change, the FD must be an FD in the
current process's FD table (aka "pre-shuffle").
To be compatible with both versions this CL introduces a new boolean
"CttyFdIsPostShuffle" which indicates whether a pre- or post-shuffle FD should
be provided. We use build tags to chose the correct one.
1: https://go-review.googlesource.com/c/go/+/178919/
PiperOrigin-RevId: 255015303
Containerd uses the last error message sent to the log to
print as failure cause for create/exec. This required a
few changes in the logging logic for runsc:
- cmd.Errorf/Fatalf: now writes a message with 'error'
level to containerd log, in addition to stderr and
debug logs, like before.
- log.Infof/Warningf/Fatalf: are not sent to containerd
log anymore. They are mostly used for debugging and not
useful to containerd. In most cases, --debug-log is
enabled and this avoids the logs messages from being
duplicated.
- stderr is not used as default log destination anymore.
Some commands assume stdio is for the container/process
running inside the sandbox and it's better to never use
it for logging. By default, logs are supressed now.
PiperOrigin-RevId: 251881815
clearStatus was added to allow detached execution to wait
on the exec'd process and retrieve its exit status. However,
it's not currently used. Both docker and gvisor-containerd-shim
wait on the "shim" process and retrieve the exit status from
there. We could change gvisor-containerd-shim to use waits, but
it will end up also consuming a process for the wait, which is
similar to having the shim process.
Closes#234
PiperOrigin-RevId: 251349490
Based on the guidelines at
https://opensource.google.com/docs/releasing/authors/.
1. $ rg -l "Google LLC" | xargs sed -i 's/Google LLC.*/The gVisor Authors./'
2. Manual fixup of "Google Inc" references.
3. Add AUTHORS file. Authors may request to be added to this file.
4. Point netstack AUTHORS to gVisor AUTHORS. Drop CONTRIBUTORS.
Fixes#209
PiperOrigin-RevId: 245823212
Change-Id: I64530b24ad021a7d683137459cafc510f5ee1de9
Removed "error" and "failed to" prefix that don't add value
from messages. Adjusted a few other messages. In particular,
when the container fail to start, the message returned is easier
for humans to read:
$ docker run --rm --runtime=runsc alpine foobar
docker: Error response from daemon: OCI runtime start failed: <path> did not terminate sucessfully: starting container: starting root container [foobar]: starting sandbox: searching for executable "foobar", cwd: "/", $PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin": no such file or directory
Closes#77
PiperOrigin-RevId: 230022798
Change-Id: I83339017c70dae09e4f9f8e0ea2e554c4d5d5cd1
Terminal support in runsc relies on host tty file descriptors that are imported
into the sandbox. Application tty ioctls are sent directly to the host fd.
However, those host tty ioctls are associated in the host kernel with a host
process (in this case runsc), and the host kernel intercepts job control
characters like ^C and send signals to the host process. Thus, typing ^C into a
"runsc exec" shell will send a SIGINT to the runsc process.
This change makes "runsc exec" handle all signals, and forward them into the
sandbox via the "ContainerSignal" urpc method. Since the "runsc exec" is
associated with a particular container process in the sandbox, the signal must
be associated with the same container process.
One big difficulty is that the signal should not necessarily be sent to the
sandbox process started by "exec", but instead must be sent to the foreground
process group for the tty. For example, we may exec "bash", and from bash call
"sleep 100". A ^C at this point should SIGINT sleep, not bash.
To handle this, tty files inside the sandbox must keep track of their
foreground process group, which is set/get via ioctls. When an incoming
ContainerSignal urpc comes in, we look up the foreground process group via the
tty file. Unfortunately, this means we have to expose and cache the tty file in
the Loader.
Note that "runsc exec" now handles signals properly, but "runs run" does not.
That will come in a later CL, as this one is complex enough already.
Example:
root@:/usr/local/apache2# sleep 100
^C
root@:/usr/local/apache2# sleep 100
^Z
[1]+ Stopped sleep 100
root@:/usr/local/apache2# fg
sleep 100
^C
root@:/usr/local/apache2#
PiperOrigin-RevId: 215334554
Change-Id: I53cdce39653027908510a5ba8d08c49f9cf24f39
This makes `runsc wait` behave more like waitpid()/wait4() in that:
- Once a process has run to completion, you can wait on it and get its exit
code.
- Processes not waited on will consume memory (like a zombie process)
PiperOrigin-RevId: 213358916
Change-Id: I5b5eca41ce71eea68e447380df8c38361a4d1558
This is different from the existing -pid-file flag, which saves a host pid.
PiperOrigin-RevId: 212713968
Change-Id: I2c486de8dd5cfd9b923fb0970165ef7c5fc597f0
When no capabilities are specified in exec, use the
container's capabilities to match runc's behavior.
PiperOrigin-RevId: 211735186
Change-Id: Icd372ed64410c81144eae94f432dffc9fe3a86ce
Etienne Perot