This moves the initialization of `cpuid.hostFeatureSet` out of package-level
`init` and instead moves it to an explicit `cpuid.Initialize()` function.
On AMD64, this saves about 512KiB of heap memory that would otherwise always
be live.
PiperOrigin-RevId: 514896323
For all trivial and zero frame-sized functions, we now require an explicit
NOFRAME annotation as the heuristic has changed. See go.dev/cl/466316.
Most of these functions do not *require* NOFRAME, but this change encodes
the existing behavior in order to avoid accidental bugs or regressions.
PiperOrigin-RevId: 513946210
This allows fact information to be validated in the underlying source files,
but requires us to explicitly maintain this in appropriate version-tagged, and
architecture-tagged files. This is more explicit and safer.
This mechanism uses a special regular expression for matching a +checkconst
stanza to validate constant values, sizes and offsets. This applies to both Go
source files and assembly files.
PiperOrigin-RevId: 511867507
Unexported constants can end up unaddressable in objectpath and thus their
object facts get lost on fact import. Work around this by adding a
package-level fact which is a map of all constants to their value.
Remove the Value facts because it is confusing to have two ways to access the
same thing (especially when one only works sometimes).
PiperOrigin-RevId: 505153453
It is similar with a16cf1420a ("Go 1.17 support for the KVM platform") that
fixed the same problem for x86. Go 1.18 adds the support of the register-based
calling convention on aarch64.
Signed-off-by: Andrei Vagin <[email protected]>
There must be no space in "//go:nosplit". This probably never
caused problems because the function is so simple it is almost
certainly inlined.
PiperOrigin-RevId: 464823961
We don't expect that runsc uses MPX . MPX was deprecated and its support
has been removed from the Linux Kernel.
We have seen when the host XCR has these bits, but a KVM VM doesn't
support MPX and xsetbv generates #GP.
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/gvisor/pull/7366 from avagin:gvisor-vs-mpx 9cd6a27048cbcd0c01dbecd7697e2c4619b29f28
PiperOrigin-RevId: 438922868
We have to calculate offsets in the CPU structure,
but we calculate offsets in the CPUArchState structure.
Fixes: dedef233cd ("Refactor nogo and provide facts render.")
PiperOrigin-RevId: 427336023
SMAP can be temporarily disabled for explicit memory accesses by setting
the EFLAGS.AC (Alignment Check) flag. But in out case, EFLAGS.AC is in
KernelFlagsClear.
Sentry never reads/writes app memory from app address space.
PiperOrigin-RevId: 423967732
This change makes the core nogo package less of a "catch all", and splits
functionality into multiple packages. Instead of separate binaries for each
function, a single "cli" package is added with subcommands, and the core
starlark wrappers are also refactored to minimize redundancy.
The new "cli" package also adds support for a "render" command, which
allows factors to be rendered via a Go text template. This is useful for
debugging, but also allows code generation to be updated to use this
mechanism. This eliminates the use of a QEMU wrapper for the older
arch_genrule, and allows the use of a native bazel transition to extract
facts for the appropriate generated file. In other words, the correct facts
will be rendered for generating XXX_arm64.s, even on amd64.
PiperOrigin-RevId: 422846459
ring0.Save/LoadFloatingPoint() are only usable if the caller can ensure that Go
will not clobber floating point registers before/after calling them
respectively. Due to regabig in Go 1.17, this is no longer the case; regabig
(among other things) maintains a zeroed XMM15 during ABIInternal execution,
including by zeroing it after ABI0-to-ABIInternal transitions. In
ring0.sysenter/exception, this happens in
ring0.kernelSyscall/kernelException.abi0 respectively; in
ring0.CPU.SwitchToUser, this happens after returning from
ring0.sysret/iret.abi0. Delete these functions and do floating point save/load
in assembly.
While arm64 doesn't appear to be immediately affected (so this CL permits us to
resume usage of Go 1.17), its use of Save/LoadFloatingPoint() still seems to be
incorrect for the same fundamental reason (Go code can't sanely assume what
registers the Go compiler will or won't use) and should be fixed eventually.
PiperOrigin-RevId: 401895658
We install seccomp rules so that the SIGSYS signal is generated for
each mmap system call. Then our signal handler executes the real mmap
syscall and if a new regions is created, it maps it to the guest.
Signed-off-by: Andrei Vagin <[email protected]>
CR0.NE enables internal x87 floating point error reporting when set, else
enables PC style x87 error detection.
On AMD, the #MF exception isn't generated if CR0.NE isn't set.
PiperOrigin-RevId: 386340269
Go 1.17 adds a new register-based calling convention. While transparent for
most applications, the KVM platform needs special work in a few cases.
First of all, we need the actual address of some assembly functions, rather
than the address of a wrapper. See http://gvisor.dev/pr/5832 for complete
discussion of this.
More relevant to this CL is that ABI0-to-ABIInternal wrappers (i.e., calls from
assembly to Go) access the G via FS_BASE. The KVM quite fast-and-loose about
the Go environment, often calling into (nosplit) Go functions with
uninitialized FS_BASE.
That will no longer work in Go 1.17, so this CL changes the platform to
consistently restore FS_BASE before calling into Go code.
This CL does not affect arm64 code. Go 1.17 does not support the register-based
calling convention for arm64 (it will come in 1.18), but arm64 also does not
use a non-standard register like FS_BASE for TLS, so it may not require any
changes.
PiperOrigin-RevId: 384234305
XCR0 has to be synchronized with the host. We can call xsave from the host
context and then call xrstor from the guest context and vise versa. This means
we need to support the same set of FPU features in both contexts.
PiperOrigin-RevId: 378988281
The root table physical page has to be mapped to not fault in iret or sysret
after switching into a user address space. sysret and iret are in the upper
half that is global and so page tables of lower levels are already mapped.
Fixes#5742
PiperOrigin-RevId: 371458644
Split usermem package to help remove syserror dependency in go_marshal.
New hostarch package contains code not dependent on syserror.
PiperOrigin-RevId: 365651233
It is enough to invalidate the tlb of local vcpu in switch().
TLBI with inner-sharable will invalidate the tlb in other vcpu.
Arm64 hardware supports at least 256 pcid, so I think it's ok
to set the length of pcid pool to 128.
Signed-off-by: Robin Luk <[email protected]>
Implement basic lazy save and restore for FPSIMD registers, which only
restore FPSIMD state on el0_fpsimd_acc and save FPSIMD state in switch().
Signed-off-by: Robin Luk <[email protected]>
In order to improve the performance and stability, I reorg 2 modules slightly.
arch: no red zone on Arm64.
ring0: use stp instead of movd, and set RSV_REG_APP=R19.
Signed-off-by: Robin Luk <[email protected]>
This allows the package to serve as a general purpose ring0 support package, as
opposed to being bound to specific sentry platforms.
Updates #5039
PiperOrigin-RevId: 355220044
