wasm2go/internal/codegen/helpers/simd_asm_amd64.s at main

hanzoai/wasm2go

Fork 0

mirror of https://github.com/hanzoai/wasm2go.git synced 2026-10-30 15:06:48 +00:00

Files

T

f30ec292fd Support the wasm SIMD proposal with native asm lowering (arm64/amd64) (#40 )

* Support the wasm SIMD proposal with native asm lowering (arm64/amd64)

One revision spanning the SIMD (0xfd v128) proposal end to end: SSA
representation, lowering, scalar reference helpers, and a gcasm asm
backend that turns the hot paths into native SIMD — measured on a
full llama.cpp build transpiled to pure Go, which this makes run at
V8-class speed (prompt eval beats V8 on arm64).

Frontend and SSA:
- v128 values are [2]uint64 pairs in little-endian lane order; pure
  ops lower to CSE-able helper calls, memory ops keep statement
  order, v128.const/shuffle ride as [2]uint64 aux. The full scalar
  helper set is the reference semantics (saturation, NaN
  propagation, round-to-even, swizzle OOB, shift masking).
- Exception handling lowers to plain branches instead of
  panic/recover — a check-and-branch after each may-raise call, a
  per-try dispatch block running the catch-clause chain — covering
  the C++-shaped patterns (multi-clause catch, delegate targets,
  rethrow depths) without runtime unwinding, so EH bodies stay
  eligible for the asm backend. This replaces the closure/recover
  lowering that #39 extended; its tests are retained unchanged.
- Giant functions outline into tractable units (whole loops
  included, packed u64 boundaries for v128 values), keeping gc
  compile times and register pressure sane on llama.cpp-sized
  modules.

gcasm backend:
- SIMD helper calls splice inline as native NEON/SSE instead of
  marshalled calls; v128 SSA values travel register-to-register in
  pair form. Call-tree windows fuse into single splices with region
  invariant hoisting, bounds-check coalescing, store sinks and
  scalar-chain internalization; whole countdown loops fuse with an
  in-splice unroll lane.
- Superinstruction selection inside fused regions: 8-bit dot
  patterns (SMULL+SADALP), SDOT for full 16-byte dots, FMUL by
  element, scale-lookup addressing, inlined integer division, the
  software fp32->fp16 idiom lowered to FCVT (both the table-gather
  and the packed store form), and ggml q8_0 row/column pairing via
  the traits nrows patch with an SMMLA 2x2 tile kernel.
- Runtime CPU-feature dispatch gates spliced bodies (dotprod on
  arm64, x86-64-v2 SSE on amd64); WASM2GO_FAST_MATH opts into
  raw-SDOT/FMLA/dual-accumulator forms that trade wasm bit-exactness
  for native-build-style rounding, validated by token-level
  equivalence against native llama.cpp.
- Guard-page linear memory eliminates per-access bounds checks;
  jump tables replace the compare tree for br_table.

Tooling and CI:
- WASM2GO_F16_TABLE asserts the address of ggml's runtime-built f16
  table; the transpiler warns when the asserted address matches no
  gather site (the address moves with every data-layout shift, and a
  stale assertion silently costs ~7% decode throughput).
- The coverage gate runs with -count=1: a cached package result
  merges incomplete coverage data into the -coverpkg profile,
  deflating the total by 10+ points depending on what the runner's
  restored cache contained — the gate must never consume cached
  results.
- Coverage threshold moves to 83% for the enlarged tree; transpile
  output is deterministic and outline rejections are explained in
  the build log.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Replace environment-variable switches with typed Options; drop debug code

Configuration now flows through one channel: transpile.Options gains
OutlineMinValues, SIMDUnroll, FuseLoops, FuseLoopUnroll, F16TableAddr
and FastMath (with matching CLI flags), the gcasm backend receives
them as a Config threaded through ModuleOffsets, and no code path
branches on an environment variable anymore. Integrator entry points
(the wasmify protoc plugin, the CLI) convert their external
configuration into these options exactly once.

Deleted outright, per the no-env rule and because they were
diagnostics rather than build modes:
- every WASM2GO_NO_* kill switch and bisection knob (SSA pass
  disabling, fallback ranges, fuse-window caps, outline limits,
  inliner windows) — git history keeps them for the next hunt;
- every env-gated debug printer (fuse/inline/outline/nrc2/f16/SSA
  dumps and the near-miss tracers) and the capture-tree keepers;
- the dormant multi-exit loop structuring (opt-in, measured slower)
  and the SSA f32->f16 rewrite entry superseded by the gcasm
  peephole (its shared idiom matchers stay for the f16 store pass);
- the runtime WASM2GO_CPU_PORTABLE override in the generated CPU
  feature probes.

The nil-offsets error paths exposed by the threading get nil-safe
Config accessors, and the outline share/v128-min/inliner caps become
plain constants at their measured production values.

Behavior under default options is unchanged: transpiled output for
the non-SIMD and EH fixtures is byte-identical to the previous
commit, and the only SIMD-bundle difference is the removed env check
in the emitted cpufeat runtime files.

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Co-authored-by: Claude Fable 5 <[email protected]>

2026-08-08 23:28:14 +09:00

1747 lines

32 KiB

ArmAsm

Raw Permalink Blame History

 // Code generated by tools/gen-simd-asm/gen.py; DO NOT EDIT.
 //
 // SSE bodies for the simd_* helpers, x86-64-v2 baseline (the
 // amd64.v2 build tag guards them; GOAMD64=v1 builds use the
 // scalar fallback aliases instead).
 //go:build amd64.v2
 #include "textflag.h"
 GLOBL ·simdConst16(SB), RODATA|NOPTR, $16
 DATA ·simdConst16+0(SB)/8, $0x1010101010101010
 DATA ·simdConst16+8(SB)/8, $0x1010101010101010
 GLOBL ·simdConst2p31d(SB), RODATA|NOPTR, $16
 DATA ·simdConst2p31d+0(SB)/8, $0x41dfffffffc00000
 DATA ·simdConst2p31d+8(SB)/8, $0x41dfffffffc00000
 GLOBL ·simdConst2p31f(SB), RODATA|NOPTR, $16
 DATA ·simdConst2p31f+0(SB)/8, $0x4f0000004f000000
 DATA ·simdConst2p31f+8(SB)/8, $0x4f0000004f000000
 GLOBL ·simdConst65536f(SB), RODATA|NOPTR, $16
 DATA ·simdConst65536f+0(SB)/8, $0x4780000047800000
 DATA ·simdConst65536f+8(SB)/8, $0x4780000047800000
 GLOBL ·simdConst70(SB), RODATA|NOPTR, $16
 DATA ·simdConst70+0(SB)/8, $0x7070707070707070
 DATA ·simdConst70+8(SB)/8, $0x7070707070707070
 GLOBL ·simdConstAbs32(SB), RODATA|NOPTR, $16
 DATA ·simdConstAbs32+0(SB)/8, $0x7fffffff7fffffff
 DATA ·simdConstAbs32+8(SB)/8, $0x7fffffff7fffffff
 GLOBL ·simdConstAbs64(SB), RODATA|NOPTR, $16
 DATA ·simdConstAbs64+0(SB)/8, $0x7fffffffffffffff
 DATA ·simdConstAbs64+8(SB)/8, $0x7fffffffffffffff
 GLOBL ·simdConstI16Min(SB), RODATA|NOPTR, $16
 DATA ·simdConstI16Min+0(SB)/8, $0x8000800080008000
 DATA ·simdConstI16Min+8(SB)/8, $0x8000800080008000
 GLOBL ·simdConstMagicHi(SB), RODATA|NOPTR, $16
 DATA ·simdConstMagicHi+0(SB)/8, $0x4330000043300000
 DATA ·simdConstMagicHi+8(SB)/8, $0x4330000043300000
 GLOBL ·simdConstMagicd(SB), RODATA|NOPTR, $16
 DATA ·simdConstMagicd+0(SB)/8, $0x4330000000000000
 DATA ·simdConstMagicd+8(SB)/8, $0x4330000000000000
 GLOBL ·simdConstNib(SB), RODATA|NOPTR, $16
 DATA ·simdConstNib+0(SB)/8, $0x0f0f0f0f0f0f0f0f
 DATA ·simdConstNib+8(SB)/8, $0x0f0f0f0f0f0f0f0f
 GLOBL ·simdConstOnes16(SB), RODATA|NOPTR, $16
 DATA ·simdConstOnes16+0(SB)/8, $0x0001000100010001
 DATA ·simdConstOnes16+8(SB)/8, $0x0001000100010001
 GLOBL ·simdConstOnes8(SB), RODATA|NOPTR, $16
 DATA ·simdConstOnes8+0(SB)/8, $0x0101010101010101
 DATA ·simdConstOnes8+8(SB)/8, $0x0101010101010101
 GLOBL ·simdConstPopLUT(SB), RODATA|NOPTR, $16
 DATA ·simdConstPopLUT+0(SB)/8, $0x0302020102010100
 DATA ·simdConstPopLUT+8(SB)/8, $0x0403030203020201
 GLOBL ·simdConstSign32(SB), RODATA|NOPTR, $16
 DATA ·simdConstSign32+0(SB)/8, $0x8000000080000000
 DATA ·simdConstSign32+8(SB)/8, $0x8000000080000000
 GLOBL ·simdConstSign64(SB), RODATA|NOPTR, $16
 DATA ·simdConstSign64+0(SB)/8, $0x8000000000000000
 DATA ·simdConstSign64+8(SB)/8, $0x8000000000000000
 GLOBL ·simdConstUMaxd(SB), RODATA|NOPTR, $16
 DATA ·simdConstUMaxd+0(SB)/8, $0x41efffffffe00000
 DATA ·simdConstUMaxd+8(SB)/8, $0x41efffffffe00000
 TEXT ·simd_f32x4_abs(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstAbs32(SB), X3
 	ANDPS X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_add(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	ADDPS X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_ceil(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPS $2, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_convert_i32x4_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	CVTPL2PS X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_convert_i32x4_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU X0, X3
 	PSRLL $16, X3
 	PSLLL $16, X0
 	PSRLL $16, X0
 	CVTPL2PS X0, X0
 	CVTPL2PS X3, X3
 	MOVOU ·simdConst65536f(SB), X4
 	MULPS X4, X3
 	ADDPS X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_demote_f64x2_zero(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	CVTPD2PS X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_div(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	DIVPS X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_eq(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPS X1, X0, $0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_floor(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPS $1, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_ge(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	CMPPS X0, X3, $2
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_gt(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	CMPPS X0, X3, $1
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_le(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPS X1, X0, $2
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_lt(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPS X1, X0, $1
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_max(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	MAXPS X1, X3
 	MOVOU X1, X4
 	MAXPS X0, X4
 	MOVOU X3, X5
 	XORPS X4, X5
 	ORPS X5, X3
 	SUBPS X5, X3
 	MOVOU X3, X5
 	CMPPS X5, X5, $3
 	ORPS X5, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_min(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	MINPS X1, X3
 	MOVOU X1, X4
 	MINPS X0, X4
 	ORPS X4, X3
 	MOVOU X3, X5
 	CMPPS X5, X5, $3
 	ORPS X5, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_mul(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MULPS X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_ne(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPS X1, X0, $4
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_nearest(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPS $0, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_neg(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstSign32(SB), X3
 	XORPS X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_pmax(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	MAXPS X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_pmin(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	MINPS X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_splat(SB), NOSPLIT, $0-24
 	MOVSS x+0(FP), X0
 	SHUFPS $0, X0, X0
 	MOVOU X0, ret+8(FP)
 	RET
 TEXT ·simd_f32x4_sqrt(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	SQRTPS X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f32x4_sub(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	SUBPS X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f32x4_trunc(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPS $3, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_abs(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstAbs64(SB), X3
 	ANDPD X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_add(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	ADDPD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_ceil(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPD $2, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_convert_low_i32x4_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	CVTPL2PD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_convert_low_i32x4_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstMagicHi(SB), X3
 	PUNPCKLLQ X3, X0
 	MOVOU ·simdConstMagicd(SB), X4
 	SUBPD X4, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_div(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	DIVPD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_eq(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPD X1, X0, $0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_floor(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPD $1, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_ge(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	CMPPD X0, X3, $2
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_gt(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	CMPPD X0, X3, $1
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_le(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPD X1, X0, $2
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_lt(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPD X1, X0, $1
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_max(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	MAXPD X1, X3
 	MOVOU X1, X4
 	MAXPD X0, X4
 	MOVOU X3, X5
 	XORPD X4, X5
 	ORPD X5, X3
 	SUBPD X5, X3
 	MOVOU X3, X5
 	CMPPD X5, X5, $3
 	ORPD X5, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_min(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	MINPD X1, X3
 	MOVOU X1, X4
 	MINPD X0, X4
 	ORPD X4, X3
 	MOVOU X3, X5
 	CMPPD X5, X5, $3
 	ORPD X5, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_mul(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MULPD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_ne(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	CMPPD X1, X0, $4
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_nearest(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPD $0, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_neg(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstSign64(SB), X3
 	XORPD X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_pmax(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	MAXPD X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_pmin(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	MINPD X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_promote_low_f32x4(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	CVTPS2PD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_splat(SB), NOSPLIT, $0-24
 	MOVSD x+0(FP), X0
 	UNPCKLPD X0, X0
 	MOVOU X0, ret+8(FP)
 	RET
 TEXT ·simd_f64x2_sqrt(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	SQRTPD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_f64x2_sub(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	SUBPD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_f64x2_trunc(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	ROUNDPD $3, X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_abs(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PABSW X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_add(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_add_sat_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDSW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_add_sat_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDUSW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_all_true(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PCMPEQW X0, X3
 	PMOVMSKB X3, AX
 	TESTL AX, AX
 	SETEQ AX
 	MOVBLZX AX, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_avgr_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PAVGW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_bitmask(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	PACKSSWB X0, X0
 	PMOVMSKB X0, AX
 	ANDL $0xff, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_eq(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_extadd_pairwise_i8x16_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstOnes8(SB), X3
 	PMADDUBSW X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_extadd_pairwise_i8x16_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstOnes8(SB), X3
 	PMADDUBSW X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_extend_high_i8x16_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PSRLO $8, X0
 	PMOVSXBW X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_extend_high_i8x16_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PSRLO $8, X0
 	PMOVZXBW X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_extend_low_i8x16_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PMOVSXBW X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_extend_low_i8x16_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PMOVZXBW X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_extmul_high_i8x16_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSRLO $8, X0
 	PSRLO $8, X1
 	PMOVSXBW X0, X0
 	PMOVSXBW X1, X1
 	PMULLW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_extmul_high_i8x16_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSRLO $8, X0
 	PSRLO $8, X1
 	PMOVZXBW X0, X0
 	PMOVZXBW X1, X1
 	PMULLW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_extmul_low_i8x16_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMOVSXBW X0, X0
 	PMOVSXBW X1, X1
 	PMULLW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_extmul_low_i8x16_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMOVZXBW X0, X0
 	PMOVZXBW X1, X1
 	PMULLW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_ge_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTW X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_ge_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMAXUW X1, X3
 	PCMPEQW X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_gt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_gt_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMINUW X1, X3
 	PCMPEQW X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_le_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTW X1, X0
 	PCMPEQL X4, X4
 	PXOR X4, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_le_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMINUW X1, X3
 	PCMPEQW X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_lt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTW X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_lt_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMAXUW X1, X3
 	PCMPEQW X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_max_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMAXSW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_max_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMAXUW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_min_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMINSW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_min_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMINUW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_mul(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMULLW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_narrow_i32x4_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PACKSSLW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_narrow_i32x4_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PACKUSDW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_ne(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQW X1, X0
 	PCMPEQL X3, X3
 	PXOR X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_neg(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PSUBW X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i16x8_q15mulr_sat_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU ·simdConstI16Min(SB), X3
 	MOVOU X0, X4
 	PCMPEQW X3, X4
 	MOVOU X1, X5
 	PCMPEQW X3, X5
 	PAND X5, X4
 	PMULHRSW X1, X0
 	PXOR X4, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_shl(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $15, AX
 	MOVQ AX, X3
 	PSLLW X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i16x8_shr_s(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $15, AX
 	MOVQ AX, X3
 	PSRAW X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i16x8_shr_u(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $15, AX
 	MOVQ AX, X3
 	PSRLW X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i16x8_splat(SB), NOSPLIT, $0-24
 	MOVL x+0(FP), AX
 	MOVQ AX, X0
 	PSHUFLW $0, X0, X0
 	PSHUFD $0, X0, X0
 	MOVOU X0, ret+8(FP)
 	RET
 TEXT ·simd_i16x8_sub(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_sub_sat_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBSW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i16x8_sub_sat_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBUSW X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_abs(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PABSD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_add(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDL X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_all_true(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PCMPEQL X0, X3
 	PMOVMSKB X3, AX
 	TESTL AX, AX
 	SETEQ AX
 	MOVBLZX AX, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_bitmask(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	MOVMSKPS X0, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_dot_i16x8_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMADDWL X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_eq(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQL X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_extadd_pairwise_i16x8_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstOnes16(SB), X3
 	PMADDWL X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_extadd_pairwise_i16x8_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU X0, X3
 	PSRLL $16, X3
 	PSLLL $16, X0
 	PSRLL $16, X0
 	PADDD X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_extend_high_i16x8_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PSRLO $8, X0
 	PMOVSXWD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_extend_high_i16x8_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PSRLO $8, X0
 	PMOVZXWD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_extend_low_i16x8_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PMOVSXWD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_extend_low_i16x8_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PMOVZXWD X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_extmul_high_i16x8_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSRLO $8, X0
 	PSRLO $8, X1
 	PMOVSXWD X0, X0
 	PMOVSXWD X1, X1
 	PMULLD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_extmul_high_i16x8_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSRLO $8, X0
 	PSRLO $8, X1
 	PMOVZXWD X0, X0
 	PMOVZXWD X1, X1
 	PMULLD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_extmul_low_i16x8_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMOVSXWD X0, X0
 	PMOVSXWD X1, X1
 	PMULLD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_extmul_low_i16x8_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMOVZXWD X0, X0
 	PMOVZXWD X1, X1
 	PMULLD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_ge_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTL X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_ge_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMAXUD X1, X3
 	PCMPEQL X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_gt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTL X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_gt_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMINUD X1, X3
 	PCMPEQL X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_le_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTL X1, X0
 	PCMPEQL X4, X4
 	PXOR X4, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_le_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMINUD X1, X3
 	PCMPEQL X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_lt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTL X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_lt_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMAXUD X1, X3
 	PCMPEQL X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_max_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMAXSD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_max_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMAXUD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_min_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMINSD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_min_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMINUD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_mul(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMULLD X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_ne(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQL X1, X0
 	PCMPEQL X3, X3
 	PXOR X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_neg(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PSUBL X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_shl(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $31, AX
 	MOVQ AX, X3
 	PSLLL X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i32x4_shr_s(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $31, AX
 	MOVQ AX, X3
 	PSRAL X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i32x4_shr_u(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $31, AX
 	MOVQ AX, X3
 	PSRLL X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i32x4_splat(SB), NOSPLIT, $0-24
 	MOVL x+0(FP), AX
 	MOVQ AX, X0
 	PSHUFD $0, X0, X0
 	MOVOU X0, ret+8(FP)
 	RET
 TEXT ·simd_i32x4_sub(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBL X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i32x4_trunc_sat_f32x4_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU X0, X3
 	CMPPS X3, X3, $7
 	ANDPS X3, X0
 	MOVOU ·simdConst2p31f(SB), X4
 	MOVOU X0, X5
 	CMPPS X4, X5, $5
 	CVTTPS2PL X0, X0
 	PXOR X5, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_trunc_sat_f32x4_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	MAXPS X3, X0
 	MOVOU ·simdConst2p31f(SB), X4
 	MOVOU X0, X5
 	SUBPS X4, X5
 	MOVOU X5, X6
 	CMPPS X4, X6, $5
 	CVTTPS2PL X5, X5
 	PXOR X6, X5
 	PXOR X7, X7
 	PMAXSD X7, X5
 	CVTTPS2PL X0, X0
 	PADDD X5, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_trunc_sat_f64x2_s_zero(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU X0, X3
 	CMPPD X3, X3, $7
 	ANDPD X3, X0
 	MOVOU ·simdConst2p31d(SB), X4
 	MINPD X4, X0
 	CVTTPD2PL X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i32x4_trunc_sat_f64x2_u_zero(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	MAXPD X3, X0
 	MOVOU ·simdConstUMaxd(SB), X4
 	MINPD X4, X0
 	ROUNDPD $3, X0, X0
 	MOVOU ·simdConstMagicd(SB), X4
 	ADDPD X4, X0
 	SHUFPS $0x88, X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_abs(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PCMPGTQ X0, X3
 	PXOR X3, X0
 	PSUBQ X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_add(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_all_true(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PCMPEQQ X0, X3
 	PMOVMSKB X3, AX
 	TESTL AX, AX
 	SETEQ AX
 	MOVBLZX AX, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_bitmask(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	MOVMSKPD X0, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_eq(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_extend_high_i32x4_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PSRLO $8, X0
 	PMOVSXDQ X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_extend_high_i32x4_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PSRLO $8, X0
 	PMOVZXDQ X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_extend_low_i32x4_s(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PMOVSXDQ X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_extend_low_i32x4_u(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PMOVZXDQ X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_extmul_high_i32x4_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSHUFD $0xfa, X0, X0
 	PSHUFD $0xfa, X1, X1
 	PMULDQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_extmul_high_i32x4_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSHUFD $0xfa, X0, X0
 	PSHUFD $0xfa, X1, X1
 	PMULULQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_extmul_low_i32x4_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSHUFD $0x50, X0, X0
 	PSHUFD $0x50, X1, X1
 	PMULDQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_extmul_low_i32x4_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSHUFD $0x50, X0, X0
 	PSHUFD $0x50, X1, X1
 	PMULULQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_ge_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTQ X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_gt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_le_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTQ X1, X0
 	PCMPEQL X4, X4
 	PXOR X4, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_lt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTQ X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_ne(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQQ X1, X0
 	PCMPEQL X3, X3
 	PXOR X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i64x2_neg(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PSUBQ X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i64x2_shl(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $63, AX
 	MOVQ AX, X3
 	PSLLQ X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i64x2_shr_u(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $63, AX
 	MOVQ AX, X3
 	PSRLQ X3, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i64x2_splat(SB), NOSPLIT, $0-24
 	MOVQ x+0(FP), AX
 	MOVQ AX, X0
 	PSHUFD $0x44, X0, X0
 	MOVOU X0, ret+8(FP)
 	RET
 TEXT ·simd_i64x2_sub(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBQ X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_abs(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PABSB X0, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i8x16_add(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_add_sat_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDSB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_add_sat_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PADDUSB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_all_true(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PCMPEQB X0, X3
 	PMOVMSKB X3, AX
 	TESTL AX, AX
 	SETEQ AX
 	MOVBLZX AX, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i8x16_avgr_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PAVGB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_bitmask(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	PMOVMSKB X0, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_i8x16_eq(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_ge_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTB X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_ge_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMAXUB X1, X3
 	PCMPEQB X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_gt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_gt_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMINUB X1, X3
 	PCMPEQB X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_le_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPGTB X1, X0
 	PCMPEQL X4, X4
 	PXOR X4, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_le_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMINUB X1, X3
 	PCMPEQB X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_lt_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PCMPGTB X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_lt_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X0, X3
 	PMAXUB X1, X3
 	PCMPEQB X0, X3
 	PCMPEQL X4, X4
 	PXOR X4, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_max_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMAXSB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_max_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMAXUB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_min_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMINSB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_min_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PMINUB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_narrow_i16x8_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PACKSSWB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_narrow_i16x8_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PACKUSWB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_ne(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PCMPEQB X1, X0
 	PCMPEQL X3, X3
 	PXOR X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_neg(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PSUBB X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i8x16_popcnt(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	MOVOU ·simdConstPopLUT(SB), X3
 	MOVOU ·simdConstNib(SB), X4
 	MOVOU X0, X5
 	PAND X4, X0
 	PSRLW $4, X5
 	PAND X4, X5
 	MOVOU X3, X6
 	PSHUFB X0, X6
 	PSHUFB X5, X3
 	PADDB X6, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_i8x16_shl(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $7, AX
 	MOVQ AX, X3
 	MOVL AX, CX
 	MOVL $0xff, AX
 	SHLL CX, AX
 	ANDL $0xff, AX
 	IMULL $0x01010101, AX
 	MOVQ AX, X4
 	PSHUFD $0, X4, X4
 	PSLLW X3, X0
 	PAND X4, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i8x16_shr_s(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $7, AX
 	ADDL $8, AX
 	MOVQ AX, X3
 	MOVOU X0, X5
 	PUNPCKLBW X0, X0
 	PSRAW X3, X0
 	PUNPCKHBW X5, X5
 	PSRAW X3, X5
 	PACKSSWB X5, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i8x16_shr_u(SB), NOSPLIT, $0-40
 	MOVOU a+0(FP), X0
 	MOVL s+16(FP), AX
 	ANDL $7, AX
 	MOVQ AX, X3
 	MOVL AX, CX
 	MOVL $0xff, AX
 	SHRL CX, AX
 	IMULL $0x01010101, AX
 	MOVQ AX, X4
 	PSHUFD $0, X4, X4
 	PSRLW X3, X0
 	PAND X4, X0
 	MOVOU X0, ret+24(FP)
 	RET
 TEXT ·simd_i8x16_shuffle(SB), NOSPLIT, $0-64
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU pat+32(FP), X2
 	MOVOU ·simdConst70(SB), X3
 	MOVOU X2, X4
 	PADDUSB X3, X4
 	PSHUFB X4, X0
 	MOVOU ·simdConst16(SB), X5
 	PSUBB X5, X2
 	PADDUSB X3, X2
 	PSHUFB X2, X1
 	POR X1, X0
 	MOVOU X0, ret+48(FP)
 	RET
 TEXT ·simd_i8x16_splat(SB), NOSPLIT, $0-24
 	MOVL x+0(FP), AX
 	MOVQ AX, X0
 	PXOR X3, X3
 	PSHUFB X3, X0
 	MOVOU X0, ret+8(FP)
 	RET
 TEXT ·simd_i8x16_sub(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_sub_sat_s(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBSB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_sub_sat_u(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PSUBUSB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_i8x16_swizzle(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU s+16(FP), X1
 	MOVOU ·simdConst70(SB), X3
 	PADDUSB X3, X1
 	PSHUFB X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_v128_and(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PAND X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_v128_andnot(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU X1, X3
 	PANDN X0, X3
 	MOVOU X3, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_v128_any_true(SB), NOSPLIT, $0-20
 	MOVOU a+0(FP), X0
 	PXOR X3, X3
 	PCMPEQB X0, X3
 	PMOVMSKB X3, AX
 	CMPL AX, $0xffff
 	SETNE AX
 	MOVBLZX AX, AX
 	MOVL AX, ret+16(FP)
 	RET
 TEXT ·simd_v128_bitselect(SB), NOSPLIT, $0-64
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	MOVOU c+32(FP), X2
 	PAND X2, X0
 	PANDN X1, X2
 	POR X2, X0
 	MOVOU X0, ret+48(FP)
 	RET
 TEXT ·simd_v128_not(SB), NOSPLIT, $0-32
 	MOVOU a+0(FP), X0
 	PCMPEQL X3, X3
 	PXOR X3, X0
 	MOVOU X0, ret+16(FP)
 	RET
 TEXT ·simd_v128_or(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	POR X1, X0
 	MOVOU X0, ret+32(FP)
 	RET
 TEXT ·simd_v128_xor(SB), NOSPLIT, $0-48
 	MOVOU a+0(FP), X0
 	MOVOU b+16(FP), X1
 	PXOR X1, X0
 	MOVOU X0, ret+32(FP)
 	RET

1747 lines 32 KiB ArmAsm Raw Permalink Blame History

1747 lines

32 KiB

ArmAsm

Raw Permalink Blame History