Host networking did not have checkpoint/restore support. A sandbox running with `--network=host` could not be checkpointed because `hostinet.Stack` was not savable, and restore did not define what should happen to open host sockets.
This change makes the hostinet stack savable and defines that restore contract. Host socket fds cannot cross a checkpoint boundary, so `Socket.fd` is tagged `state:"nosave"` and save does not touch host sockets. With checkpoint `--leave-running`, the original sandbox keeps serving traffic on its existing connections. If save fails, the sandbox is left unharmed.
Without `--leave-running`, the sandbox exits after save and the host kernel closes its sockets. That sends `FIN` or `RST` so remote peers do not hang on half-open connections.
In the restored sandbox, `afterLoad` sets each socket fd to `-1`. Host syscalls then fail with `EBADF`. `Readiness` reports hangup and error so `epoll` waiters wake immediately, and tasks that were blocked in I/O see `EBADF` when their syscall restarts. `State()` reports no TCP state for these sockets rather than logging a warning on every `/proc/net/tcp` read. Applications are expected to reconnect after restore.
The stack's host-derived state is not saved. This includes `/proc/net/dev`, `/proc/net/snmp`, TCP buffer sizes, and allowed socket types. During restore, runsc configures a fresh hostinet stack before seccomp filters are installed. `ReplaceConfig` then copies the destination host configuration into the deserialized stack and transfers ownership of the fresh proc net handles. The restored sandbox therefore uses the destination host configuration and limits.
A hostinet checkpoint contains no netstack state and a netstack checkpoint contains no hostinet state, so the checkpoint now records the network type in its metadata. Restore fails with a clear error when the network stack kind differs between checkpoint and restore, instead of panicking inside `ReplaceConfig` during kernel load. Sandbox and none networking both use netstack and remain interchangeable. Checkpoints taken before the network type was recorded skip this check.
The hostinet `save_resume` syscall tests are enabled because save does not touch host sockets. The save and restore syscall tests remain excluded because they expect sockets to keep working after restore.
Container tests cover restored socket behavior for TCP, UDP, `epoll`, and a blocked accept. They also cover remote peer close after checkpoint, continued service after checkpoint `--leave-running`, and the error when restoring a host networking checkpoint with sandbox networking. The focused hostinet unit tests and the hostinet `save_resume` syscall tests pass on `x86_64`. The checkpoint/restore user guide documents the restored socket behavior.
Fixes#6243
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/gvisor/pull/13404 from shayonj:hostinet-save-restore-v2 734480ee348d774346ec024b6483b89ea0a91adb
PiperOrigin-RevId: 950854464
This change refactors the network configuration flow during sandbox restore
and defers the creation of network links and routes in runsc to run after
seccomp filters are installed.
- Replaced ReplaceConfig with ResetConfig in inet.Stack. Introduced
inet.NetworkArgs to configure the stack during restore, updating k.LoadFrom
to use it.
- Renamed CreateLinksAndRoutes URPC to SetNetworkArgs, which now configures
host sockets before seccomp is installed, deferring link/route creation to run
after seccomp.
- Fixed S/R support by saving and restoring limit/burst state.
- Minor seccomp filter adjustments and test updates.
PiperOrigin-RevId: 945995430
Previously, during sandbox restore the network configuration was restored by
creating a new network stack (which has all the config scraped from the host)
and the replacing this new config such as NICs, routes and IP addresses from
the new stack to the loaded/restored stack.
This change refactors the process by introducing a new `inet.NetworkArgs`
interface that abstracts how the network stack is configured. The `Loader` now
implements this interface and configures the loaded stack directly using the
configuration scraped from the host during restore. `Kernel.LoadFrom` accepts
this interface (instead of stack), allowing it to configure the restored stack
using either the Loader's configuration.
PiperOrigin-RevId: 941330876
Go map iteration randomness causes network interface enumeration
(getifaddrs, RTM_GETADDR, RTM_GETLINK, /proc/net/*, SIOCGIFCONF)
to return nondeterministic order, breaking applications like
asciidoctor that expect stable ordering across repeated calls.
Since gVisor allocates NIC IDs monotonically (like Linux ifindex),
sorting by NICID is equivalent to registration order. Add
InterfaceIDs() to inet.Stack so all consumer paths iterate in
deterministic ascending-ID order without ad-hoc sorting at each
call site.
Signed-off-by: Tan Yifeng <[email protected]>
When using hostinet (network=host), /proc/net/dev and /proc/net/snmp
may not be available inside the sandbox. Previously Statistics() returned
an error when these files were not opened, which propagated up and caused
the entire event/metrics collection to fail, breaking kubectl top pod.
Return zero-valued stats instead of an error when the proc files are
unavailable, as this is expected in hostinet mode. Real errors like
parse failures still propagate.
Fixes#12723
Previously, external connections were terminated during both restore and
resume operations, causing a reconnection after every resume. By limiting
termination to the restore only, connections now remain active during resumes
as intended.
Added tests to verify that resume does not terminate the connection.
PiperOrigin-RevId: 868830577
Only netstack supports sending link events, requests to join multicast
groups for netlink sockets continue to be denied when hostinet is in use.
connect() and sendmsg() to RTMGRP_LINK still fails.
PiperOrigin-RevId: 831688843
During restore there will be new NICs and routes created based on the network
config. Close these old NICs and routes in beforeSave() only for save/restore.
For save/resume, do not close them.
As new link endpoints are created during restore, there is no need to start
the associated old processor goroutines in afterLoad. Start them only for
save/resume.
PiperOrigin-RevId: 804994281
- Added a new Stats() method in inet.Stack to get the saved stats
during restore.
- Mark stack.nic, tcpip.Route and stack.addressState structs as "nosave".
These fields should not be saved because the IP addresses and routes can
change during restore and new configuration of routes and IP addresses will be
extracted from the restore spec and initialized in the saved stack.
- Changes in Restore() method in icmp, udp, tcp, packet and raw endpoint files
to support save restore of these endpoints. These changes are flag guarded by
the TESTONLY-save-restore-netstack flag.
PiperOrigin-RevId: 707639274
This CL restores the listening connections when netstack s/r is enabled.
The changes include:
- New method as a workaround to replace the new routes and nics to the loaded
stack after restore.
- New Restore() for transport layer protocols to restore the protocol level
background workers.
- Adds afterLoad() method for fdbased processors.
- Adds a test to verify listening connection is restored after checkpointing
with netstack s/r enabled.
- Few other changes to save restore fields to enable netstack s/r.
PiperOrigin-RevId: 698453124
- Adds a new flag which will enable netstack s/r. When the flag is not enabled,
there is no change in the existing behavior. The flag will be enabled only in
tests to verify the s/r functionality of netstack.
- Some additional fields in netstack were causing panic when netstack is
save/restored. Such fields are marked as 'save'/'nosave' accordingly to resolve
the panic.
PiperOrigin-RevId: 668566657
The NEWLINK commands contains many properties and here is no reason to have
another abstract interface between the Sentry and network stacks.
PiperOrigin-RevId: 627916629
Resume method resumes the endpoints after save which is required for
save/resume. This method resumes the endpoint states which were frozen during
save.
PiperOrigin-RevId: 615467468
The existing Resume method in netstack is doing the work of Restore. This
method does not resume endpoints as the resumable endpoints are only stored
during the Restore of the sandbox, rename the method appropriately.
PiperOrigin-RevId: 613320887
And enable all of the netlink tests that depend on this functionality.
Specifially, this CL implements hostinet.stack:
* .RemoveInterface()
* .AddInterfaceAddr()
* .RemoveInterfaceAddr()
Applications use netlink messages to call into these methods. For hostinet, we
use netlink messages to the host to implement these methods.
Note that the netlink messages are not simple passed from the application to
the host -- they are parsed first and only carefully crafted messages are sent
to the host.
This CL required a lot of refactoring to the existing netlink code in the
hostinet package, as well as a bunch of new functionality. All of that code has
been collected in hostinet/netlink.go
I also changed how we handle CAP_NET_ADMIN and CAP_NET_RAW in tests, since we
must drop those capabilities inside the sandbox if we don't have them on the
host. The new solution is cleaner and does not rely on google-specific env
variables.
PiperOrigin-RevId: 522685544
Previously we were initializing these at startup and they could never change.
Now they are read dynamically every time they are requested. This is a
pre-requisite for implementing methods that mutate the interfaces
(stack.AddInterface()).
PiperOrigin-RevId: 522109098
No functional changes. Refactored some methods into smaller chunks for easy
reuse. Got rid of some needlessly exported methods.
PiperOrigin-RevId: 520748526
This change removes all endpoint goroutines and all TCP processing is now done
inline in the TCP processor loop. TCP timers directly invoke handlers as
required rather than assert a waker.
UnlockUser is also simplified to just queue the endpoint to the processor
instead of trying to process segments inline. This allows us to centralize logic
for TCP state handling in the processor. This potentially could involve an extra
wakeup but now that endpoint goroutines do not exist this is not such a big
concern as in case of busy servers the processor goroutines will already be
running anyway.
This change also allows us to clean up S/R as now restoring a TCP endpoint does
not require restarting a goroutine and moving it to the right logical point but
only requires that we restart any timers that may have been running when the
save was done and restore any port bindings as required.
Endpoint.Release is now removed in favor of Endpoint.Abort by using Abort in
places where we use Endpoint.Release.
Updates #231
PiperOrigin-RevId: 442673015
Change marshal.Marshallable method signatures to return the remaining buffer.
This makes it easier to implement these method manually. Without this, we would
have to manually do buffer shifting which is error prone.
tools/go_marshal/test:benchmark test does not show change in performance.
Additionally fixed some marshalling bugs in fsimpl/fuse.
Updated multiple callpoints to get rid of redundant slice indexing work and
simplified code using this new signature.
Updates #6450
PiperOrigin-RevId: 407857019
This change will allow us to remove the default link in a packetimpact test so
we can reduce indeterministic behaviors as required in https://fxbug.dev/78430.
This will also help with testing #1388.
Updates #578, #1388.
PiperOrigin-RevId: 387896847
Update all instances of the above errors to the faster linuxerr implementation.
With the temporary linuxerr.Equals(), no logical changes are made.
PiperOrigin-RevId: 382306655
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt:
/proc/sys/net/ipv4/* Variables:
ip_forward - BOOLEAN
0 - disabled (default)
not 0 - enabled
Forward Packets between interfaces.
This variable is special, its change resets all configuration
parameters to their default state (RFC1122 for hosts, RFC1812
for routers)
/proc/sys/net/ipv4/ip_forward only does work when its value is changed
and always returns the last written value. The last written value may
not reflect the current state of the netstack (e.g. when `ip_forward`
was written a value of "1" then disable forwarding on an interface)
so there is no need for sentry to probe netstack to get the current
forwarding state of interfaces.
```
~$ cat /proc/sys/net/ipv4/ip_forward
0
~$ sudo bash -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
~$ cat /proc/sys/net/ipv4/ip_forward
1
~$ sudo sysctl -a | grep ipv4 | grep forward
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.eno1.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.wlp1s0.forwarding = 1
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
~$ sudo sysctl -w net.ipv4.conf.wlp1s0.forwarding=0
net.ipv4.conf.wlp1s0.forwarding = 0
~$ sudo sysctl -a | grep ipv4 | grep forward
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.eno1.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.wlp1s0.forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
~$ cat /proc/sys/net/ipv4/ip_forward
1
~$ sudo bash -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
~$ sudo sysctl -a | grep ipv4 | grep forward
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.eno1.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.wlp1s0.forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
~$ sudo bash -c "echo 0 > /proc/sys/net/ipv4/ip_forward"
~$ sudo sysctl -a | grep ipv4 | grep forward
sysctl: unable to open directory "/proc/sys/fs/binfmt_misc/"
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.eno1.forwarding = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.wlp1s0.forwarding = 0
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
~$ cat /proc/sys/net/ipv4/ip_forward
0
```
In the above example we can see that writing "1" to
/proc/sys/net/ipv4/ip_forward configures the stack to be a router (all
interfaces are configured to enable forwarding). However, if we manually
update an interace (`wlp1s0`) to not forward packets,
/proc/sys/net/ipv4/ip_forward continues to return the last written value
of "1", even though not all interfaces will forward packets.
Also note that writing the same value twice has no effect; work is
performed iff the value changes.
This change also removes the 'unset' state from sentry's ip forwarding
data structures as an 'unset' ip forwarding value is the same as leaving
forwarding disabled as the stack is always brought up with forwarding
initially disabled; disabling forwarding on a newly created stack is a
no-op.
PiperOrigin-RevId: 373853106
Split usermem package to help remove syserror dependency in go_marshal.
New hostarch package contains code not dependent on syserror.
PiperOrigin-RevId: 365651233
The syscall package has been deprecated in favor of golang.org/x/sys.
Note that syscall is still used in the following places:
- pkg/sentry/socket/hostinet/stack.go: some netlink related functionalities
are not yet available in golang.org/x/sys.
- syscall.Stat_t is still used in some places because os.FileInfo.Sys() still
returns it and not unix.Stat_t.
Updates #214
PiperOrigin-RevId: 360701387
Because the abi will depend on the core types for marshalling (usermem,
context, safemem, safecopy), these need to be flattened from the sentry
directory. These packages contain no sentry-specific details.
PiperOrigin-RevId: 291811289
In the future this will replace DanglingEndpoints. DanglingEndpoints must be
kept for now due to issues with save/restore.
This is arguably a cleaner design and allows the stack to know which transport
endpoints might still be using its link endpoints.
Updates #837
PiperOrigin-RevId: 277386633
