hanzoai/wasm2go
2
0
Fork 0
mirror of https://github.com/hanzoai/wasm2go.git synced 2026-10-30 15:06:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
wasm2go/internal/codegen/helpers/simd_asm_arm64.s
T
Masaaki GoshimaGitHubClaude Fable 5
 f30ec292fd Support the wasm SIMD proposal with native asm lowering (arm64/amd64) (#40) 
* Support the wasm SIMD proposal with native asm lowering (arm64/amd64)

One revision spanning the SIMD (0xfd v128) proposal end to end: SSA
representation, lowering, scalar reference helpers, and a gcasm asm
backend that turns the hot paths into native SIMD — measured on a
full llama.cpp build transpiled to pure Go, which this makes run at
V8-class speed (prompt eval beats V8 on arm64).

Frontend and SSA:
- v128 values are [2]uint64 pairs in little-endian lane order; pure
  ops lower to CSE-able helper calls, memory ops keep statement
  order, v128.const/shuffle ride as [2]uint64 aux. The full scalar
  helper set is the reference semantics (saturation, NaN
  propagation, round-to-even, swizzle OOB, shift masking).
- Exception handling lowers to plain branches instead of
  panic/recover — a check-and-branch after each may-raise call, a
  per-try dispatch block running the catch-clause chain — covering
  the C++-shaped patterns (multi-clause catch, delegate targets,
  rethrow depths) without runtime unwinding, so EH bodies stay
  eligible for the asm backend. This replaces the closure/recover
  lowering that #39 extended; its tests are retained unchanged.
- Giant functions outline into tractable units (whole loops
  included, packed u64 boundaries for v128 values), keeping gc
  compile times and register pressure sane on llama.cpp-sized
  modules.

gcasm backend:
- SIMD helper calls splice inline as native NEON/SSE instead of
  marshalled calls; v128 SSA values travel register-to-register in
  pair form. Call-tree windows fuse into single splices with region
  invariant hoisting, bounds-check coalescing, store sinks and
  scalar-chain internalization; whole countdown loops fuse with an
  in-splice unroll lane.
- Superinstruction selection inside fused regions: 8-bit dot
  patterns (SMULL+SADALP), SDOT for full 16-byte dots, FMUL by
  element, scale-lookup addressing, inlined integer division, the
  software fp32->fp16 idiom lowered to FCVT (both the table-gather
  and the packed store form), and ggml q8_0 row/column pairing via
  the traits nrows patch with an SMMLA 2x2 tile kernel.
- Runtime CPU-feature dispatch gates spliced bodies (dotprod on
  arm64, x86-64-v2 SSE on amd64); WASM2GO_FAST_MATH opts into
  raw-SDOT/FMLA/dual-accumulator forms that trade wasm bit-exactness
  for native-build-style rounding, validated by token-level
  equivalence against native llama.cpp.
- Guard-page linear memory eliminates per-access bounds checks;
  jump tables replace the compare tree for br_table.

Tooling and CI:
- WASM2GO_F16_TABLE asserts the address of ggml's runtime-built f16
  table; the transpiler warns when the asserted address matches no
  gather site (the address moves with every data-layout shift, and a
  stale assertion silently costs ~7% decode throughput).
- The coverage gate runs with -count=1: a cached package result
  merges incomplete coverage data into the -coverpkg profile,
  deflating the total by 10+ points depending on what the runner's
  restored cache contained — the gate must never consume cached
  results.
- Coverage threshold moves to 83% for the enlarged tree; transpile
  output is deterministic and outline rejections are explained in
  the build log.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Replace environment-variable switches with typed Options; drop debug code

Configuration now flows through one channel: transpile.Options gains
OutlineMinValues, SIMDUnroll, FuseLoops, FuseLoopUnroll, F16TableAddr
and FastMath (with matching CLI flags), the gcasm backend receives
them as a Config threaded through ModuleOffsets, and no code path
branches on an environment variable anymore. Integrator entry points
(the wasmify protoc plugin, the CLI) convert their external
configuration into these options exactly once.

Deleted outright, per the no-env rule and because they were
diagnostics rather than build modes:
- every WASM2GO_NO_* kill switch and bisection knob (SSA pass
  disabling, fallback ranges, fuse-window caps, outline limits,
  inliner windows) — git history keeps them for the next hunt;
- every env-gated debug printer (fuse/inline/outline/nrc2/f16/SSA
  dumps and the near-miss tracers) and the capture-tree keepers;
- the dormant multi-exit loop structuring (opt-in, measured slower)
  and the SSA f32->f16 rewrite entry superseded by the gcasm
  peephole (its shared idiom matchers stay for the f16 store pass);
- the runtime WASM2GO_CPU_PORTABLE override in the generated CPU
  feature probes.

The nil-offsets error paths exposed by the threading get nil-safe
Config accessors, and the outline share/v128-min/inliner caps become
plain constants at their measured production values.

Behavior under default options is unchanged: transpiled output for
the non-SIMD and EH fixtures is byte-identical to the previous
commit, and the only SIMD-bundle difference is the removed env check
in the emitted cpufeat runtime files.

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-08 23:28:14 +09:00

1459 lines
36 KiB
ArmAsm
Raw Permalink Blame History

// Code generated by tools/gen-simd-asm/gen.py; DO NOT EDIT.
//
// NEON bodies for the simd_* helpers. Vector instructions the Go
// assembler cannot spell are WORD-encoded (fixed registers, so the
// encodings are constants; clang is the encoder ground truth).
#include "textflag.h"
GLOBL ·simdConstBitpos8(SB), RODATA|NOPTR, $16
DATA ·simdConstBitpos8+0(SB)/8, $0x8040201008040201
DATA ·simdConstBitpos8+8(SB)/8, $0x8040201008040201
GLOBL ·simdConstBitpos16(SB), RODATA|NOPTR, $16
DATA ·simdConstBitpos16+0(SB)/8, $0x0008000400020001
DATA ·simdConstBitpos16+8(SB)/8, $0x0080004000200010
GLOBL ·simdConstBitpos32(SB), RODATA|NOPTR, $16
DATA ·simdConstBitpos32+0(SB)/8, $0x0000000200000001
DATA ·simdConstBitpos32+8(SB)/8, $0x0000000800000004
TEXT ·simd_f16x4_cvt_bits(SB), NOSPLIT|NOFRAME, $0-24
FMOVQ a+0(FP), F0
WORD $0x0e216801 // fcvtn v1.4h, v0.4s
WORD $0x4e20e402 // fcmeq v2.4s, v0.4s, v0.4s
WORD $0x6f300403 // ushr v3.4s, v0.4s, #16
WORD $0x4f042404 // movi v4.4s, #0x80, lsl #8
WORD $0x4e241c63 // and v3.16b, v3.16b, v4.16b
WORD $0x4f0327c4 // movi v4.4s, #0x7e, lsl #8
WORD $0x4ea41c63 // orr v3.16b, v3.16b, v4.16b
WORD $0x0e612842 // xtn v2.4h, v2.4s
WORD $0x0e612863 // xtn v3.4h, v3.4s
WORD $0x2e631c22 // bsl v2.8b, v1.8b, v3.8b
FMOVD F2, R0
MOVD R0, ret+16(FP)
RET
TEXT ·simd_f32x4_abs(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ea0f800 // fabs v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_add(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e21d400 // fadd v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_ceil(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ea18800 // frintp v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_convert_i32x4_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e21d800 // scvtf v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_convert_i32x4_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6e21d800 // ucvtf v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_demote_f64x2_zero(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x0e616800 // fcvtn v0.2s, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_div(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e21fc00 // fdiv v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_eq(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e21e400 // fcmeq v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_floor(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e219800 // frintm v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_ge(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e21e400 // fcmge v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_gt(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea1e400 // fcmgt v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_le(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e20e420 // fcmge v0.4s, v1.4s, v0.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_lt(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea0e420 // fcmgt v0.4s, v1.4s, v0.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_max(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e21f400 // fmax v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_min(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea1f400 // fmin v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_mul(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e21dc00 // fmul v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_ne(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e21e400 // fcmeq v0.4s, v0.4s, v1.4s
WORD $0x6e205800 // mvn v0.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_nearest(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e218800 // frintn v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_neg(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ea0f800 // fneg v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_pmax(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea0e422 // fcmgt v2.4s, v1.4s, v0.4s
WORD $0x6e601c22 // bsl v2.16b, v1.16b, v0.16b
WORD $0x4ea21c40 // mov v0.16b, v2.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_pmin(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea1e402 // fcmgt v2.4s, v0.4s, v1.4s
WORD $0x6e601c22 // bsl v2.16b, v1.16b, v0.16b
WORD $0x4ea21c40 // mov v0.16b, v2.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_splat(SB), NOSPLIT|NOFRAME, $0-24
FMOVS x+0(FP), F0
WORD $0x4e040400 // dup v0.4s, v0.s[0]
FMOVQ F0, ret+8(FP)
RET
TEXT ·simd_f32x4_sqrt(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ea1f800 // fsqrt v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f32x4_sub(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea1d400 // fsub v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f32x4_trunc(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ea19800 // frintz v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_abs(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ee0f800 // fabs v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_add(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e61d400 // fadd v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_ceil(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ee18800 // frintp v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_convert_low_i32x4_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x0f20a400 // sshll v0.2d, v0.2s, #0
WORD $0x4e61d800 // scvtf v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_convert_low_i32x4_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x2f20a400 // ushll v0.2d, v0.2s, #0
WORD $0x6e61d800 // ucvtf v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_div(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e61fc00 // fdiv v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_eq(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e61e400 // fcmeq v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_floor(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e619800 // frintm v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_ge(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e61e400 // fcmge v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_gt(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ee1e400 // fcmgt v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_le(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e60e420 // fcmge v0.2d, v1.2d, v0.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_lt(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ee0e420 // fcmgt v0.2d, v1.2d, v0.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_max(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e61f400 // fmax v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_min(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ee1f400 // fmin v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_mul(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e61dc00 // fmul v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_ne(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e61e400 // fcmeq v0.2d, v0.2d, v1.2d
WORD $0x6e205800 // mvn v0.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_nearest(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e618800 // frintn v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_neg(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ee0f800 // fneg v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_pmax(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ee0e422 // fcmgt v2.2d, v1.2d, v0.2d
WORD $0x6e601c22 // bsl v2.16b, v1.16b, v0.16b
WORD $0x4ea21c40 // mov v0.16b, v2.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_pmin(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ee1e402 // fcmgt v2.2d, v0.2d, v1.2d
WORD $0x6e601c22 // bsl v2.16b, v1.16b, v0.16b
WORD $0x4ea21c40 // mov v0.16b, v2.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_promote_low_f32x4(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x0e617800 // fcvtl v0.2d, v0.2s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_splat(SB), NOSPLIT|NOFRAME, $0-24
FMOVD x+0(FP), F0
WORD $0x4e080400 // dup v0.2d, v0.d[0]
FMOVQ F0, ret+8(FP)
RET
TEXT ·simd_f64x2_sqrt(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ee1f800 // fsqrt v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_f64x2_sub(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ee1d400 // fsub v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_f64x2_trunc(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ee19800 // frintz v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_abs(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e60b800 // abs v0.8h, v0.8h
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_add(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e618400 // add v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_add_sat_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e610c00 // sqadd v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_add_sat_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e610c00 // uqadd v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_all_true(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x6e71a801 // uminv h1, v0.8h
WORD $0x0e023c20 // umov w0, v1.h[0]
WORD $0x7100001f // cmp w0, #0
WORD $0x1a9f07e0 // cset w0, ne
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i16x8_avgr_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e611400 // urhadd v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_bitmask(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x4f110401 // sshr v1.8h, v0.8h, #15
MOVD $·simdConstBitpos16(SB), R4
FMOVQ (R4), F2
WORD $0x4e221c21 // and v1.16b, v1.16b, v2.16b
WORD $0x4e71b821 // addv h1, v1.8h
WORD $0x0e023c20 // umov w0, v1.h[0]
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i16x8_eq(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e618c00 // cmeq v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_extadd_pairwise_i8x16_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e202800 // saddlp v0.8h, v0.16b
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_extadd_pairwise_i8x16_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6e202800 // uaddlp v0.8h, v0.16b
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_extend_high_i8x16_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4f08a400 // sshll2 v0.8h, v0.16b, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_extend_high_i8x16_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6f08a400 // ushll2 v0.8h, v0.16b, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_extend_low_i8x16_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x0f08a400 // sshll v0.8h, v0.8b, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_extend_low_i8x16_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x2f08a400 // ushll v0.8h, v0.8b, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_extmul_high_i8x16_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e21c000 // smull2 v0.8h, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_extmul_high_i8x16_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e21c000 // umull2 v0.8h, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_extmul_low_i8x16_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x0e21c000 // smull v0.8h, v0.8b, v1.8b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_extmul_low_i8x16_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x2e21c000 // umull v0.8h, v0.8b, v1.8b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_ge_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e613c00 // cmge v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_ge_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e613c00 // cmhs v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_gt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e613400 // cmgt v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_gt_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e613400 // cmhi v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_le_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e603c20 // cmge v0.8h, v1.8h, v0.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_le_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e603c20 // cmhs v0.8h, v1.8h, v0.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_lt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e603420 // cmgt v0.8h, v1.8h, v0.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_lt_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e603420 // cmhi v0.8h, v1.8h, v0.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_max_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e616400 // smax v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_max_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e616400 // umax v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_min_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e616c00 // smin v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_min_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e616c00 // umin v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_mul(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e619c00 // mul v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_narrow_i32x4_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x0e614800 // sqxtn v0.4h, v0.4s
WORD $0x4e614820 // sqxtn2 v0.8h, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_narrow_i32x4_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x2e612800 // sqxtun v0.4h, v0.4s
WORD $0x6e612820 // sqxtun2 v0.8h, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_ne(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e618c00 // cmeq v0.8h, v0.8h, v1.8h
WORD $0x6e205800 // mvn v0.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_neg(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6e60b800 // neg v0.8h, v0.8h
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i16x8_q15mulr_sat_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e61b400 // sqrdmulh v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_shl(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12000c00 // and w0, w0, #15
WORD $0x4e020c01 // dup v1.8h, w0
WORD $0x6e614400 // ushl v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i16x8_shr_s(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12000c00 // and w0, w0, #15
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e020c01 // dup v1.8h, w0
WORD $0x4e614400 // sshl v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i16x8_shr_u(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12000c00 // and w0, w0, #15
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e020c01 // dup v1.8h, w0
WORD $0x6e614400 // ushl v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i16x8_splat(SB), NOSPLIT|NOFRAME, $0-24
MOVWU x+0(FP), R0
WORD $0x4e020c00 // dup v0.8h, w0
FMOVQ F0, ret+8(FP)
RET
TEXT ·simd_i16x8_sub(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e618400 // sub v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_sub_sat_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e612c00 // sqsub v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i16x8_sub_sat_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e612c00 // uqsub v0.8h, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_abs(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ea0b800 // abs v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_add(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea18400 // add v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_all_true(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x6eb1a801 // uminv s1, v0.4s
WORD $0x0e043c20 // umov w0, v1.s[0]
WORD $0x7100001f // cmp w0, #0
WORD $0x1a9f07e0 // cset w0, ne
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i32x4_bitmask(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x4f210401 // sshr v1.4s, v0.4s, #31
MOVD $·simdConstBitpos32(SB), R4
FMOVQ (R4), F2
WORD $0x4e221c21 // and v1.16b, v1.16b, v2.16b
WORD $0x4eb1b821 // addv s1, v1.4s
WORD $0x0e043c20 // umov w0, v1.s[0]
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i32x4_dot_i16x8_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x0e61c002 // smull v2.4s, v0.4h, v1.4h
WORD $0x4e61c003 // smull2 v3.4s, v0.8h, v1.8h
WORD $0x4ea3bc40 // addp v0.4s, v2.4s, v3.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_eq(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea18c00 // cmeq v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_extadd_pairwise_i16x8_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e602800 // saddlp v0.4s, v0.8h
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_extadd_pairwise_i16x8_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6e602800 // uaddlp v0.4s, v0.8h
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_extend_high_i16x8_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4f10a400 // sshll2 v0.4s, v0.8h, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_extend_high_i16x8_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6f10a400 // ushll2 v0.4s, v0.8h, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_extend_low_i16x8_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x0f10a400 // sshll v0.4s, v0.4h, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_extend_low_i16x8_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x2f10a400 // ushll v0.4s, v0.4h, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_extmul_high_i16x8_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e61c000 // smull2 v0.4s, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_extmul_high_i16x8_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e61c000 // umull2 v0.4s, v0.8h, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_extmul_low_i16x8_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x0e61c000 // smull v0.4s, v0.4h, v1.4h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_extmul_low_i16x8_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x2e61c000 // umull v0.4s, v0.4h, v1.4h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_ge_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea13c00 // cmge v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_ge_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea13c00 // cmhs v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_gt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea13400 // cmgt v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_gt_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea13400 // cmhi v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_le_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea03c20 // cmge v0.4s, v1.4s, v0.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_le_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea03c20 // cmhs v0.4s, v1.4s, v0.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_lt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea03420 // cmgt v0.4s, v1.4s, v0.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_lt_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea03420 // cmhi v0.4s, v1.4s, v0.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_max_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea16400 // smax v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_max_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea16400 // umax v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_min_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea16c00 // smin v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_min_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea16c00 // umin v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_mul(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea19c00 // mul v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_ne(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea18c00 // cmeq v0.4s, v0.4s, v1.4s
WORD $0x6e205800 // mvn v0.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_neg(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ea0b800 // neg v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_shl(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12001000 // and w0, w0, #31
WORD $0x4e040c01 // dup v1.4s, w0
WORD $0x6ea14400 // ushl v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i32x4_shr_s(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12001000 // and w0, w0, #31
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e040c01 // dup v1.4s, w0
WORD $0x4ea14400 // sshl v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i32x4_shr_u(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12001000 // and w0, w0, #31
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e040c01 // dup v1.4s, w0
WORD $0x6ea14400 // ushl v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i32x4_splat(SB), NOSPLIT|NOFRAME, $0-24
MOVWU x+0(FP), R0
WORD $0x4e040c00 // dup v0.4s, w0
FMOVQ F0, ret+8(FP)
RET
TEXT ·simd_i32x4_sub(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea18400 // sub v0.4s, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i32x4_trunc_sat_f32x4_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ea1b800 // fcvtzs v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_trunc_sat_f32x4_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ea1b800 // fcvtzu v0.4s, v0.4s
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_trunc_sat_f64x2_s_zero(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ee1b800 // fcvtzs v0.2d, v0.2d
WORD $0x0ea14800 // sqxtn v0.2s, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i32x4_trunc_sat_f64x2_u_zero(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ee1b800 // fcvtzu v0.2d, v0.2d
WORD $0x2ea14800 // uqxtn v0.2s, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i64x2_abs(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4ee0b800 // abs v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i64x2_add(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ee18400 // add v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_all_true(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x4ee09800 // cmeq v0.2d, v0.2d, #0
WORD $0x6e205800 // mvn v0.16b, v0.16b
WORD $0x6eb1a801 // uminv s1, v0.4s
WORD $0x0e043c20 // umov w0, v1.s[0]
WORD $0x7100001f // cmp w0, #0
WORD $0x1a9f07e0 // cset w0, ne
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i64x2_bitmask(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x4e083c01 // umov x1, v0.d[0]
WORD $0x4e183c02 // umov x2, v0.d[1]
WORD $0xd37ffc21 // lsr x1, x1, #63
WORD $0xd37ffc42 // lsr x2, x2, #63
WORD $0x2a020420 // orr w0, w1, w2, lsl #1
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i64x2_eq(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ee18c00 // cmeq v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_extend_high_i32x4_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4f20a400 // sshll2 v0.2d, v0.4s, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i64x2_extend_high_i32x4_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6f20a400 // ushll2 v0.2d, v0.4s, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i64x2_extend_low_i32x4_s(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x0f20a400 // sshll v0.2d, v0.2s, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i64x2_extend_low_i32x4_u(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x2f20a400 // ushll v0.2d, v0.2s, #0
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i64x2_extmul_high_i32x4_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea1c000 // smull2 v0.2d, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_extmul_high_i32x4_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ea1c000 // umull2 v0.2d, v0.4s, v1.4s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_extmul_low_i32x4_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x0ea1c000 // smull v0.2d, v0.2s, v1.2s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_extmul_low_i32x4_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x2ea1c000 // umull v0.2d, v0.2s, v1.2s
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_ge_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ee13c00 // cmge v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_gt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ee13400 // cmgt v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_le_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ee03c20 // cmge v0.2d, v1.2d, v0.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_lt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ee03420 // cmgt v0.2d, v1.2d, v0.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_ne(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ee18c00 // cmeq v0.2d, v0.2d, v1.2d
WORD $0x6e205800 // mvn v0.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i64x2_neg(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6ee0b800 // neg v0.2d, v0.2d
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i64x2_shl(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12001400 // and w0, w0, #63
WORD $0x4e080c01 // dup v1.2d, x0
WORD $0x6ee14400 // ushl v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i64x2_shr_s(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12001400 // and w0, w0, #63
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e080c01 // dup v1.2d, x0
WORD $0x4ee14400 // sshl v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i64x2_shr_u(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12001400 // and w0, w0, #63
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e080c01 // dup v1.2d, x0
WORD $0x6ee14400 // ushl v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i64x2_splat(SB), NOSPLIT|NOFRAME, $0-24
MOVD x+0(FP), R0
WORD $0x4e080c00 // dup v0.2d, x0
FMOVQ F0, ret+8(FP)
RET
TEXT ·simd_i64x2_sub(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6ee18400 // sub v0.2d, v0.2d, v1.2d
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_abs(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e20b800 // abs v0.16b, v0.16b
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i8x16_add(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e218400 // add v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_add_sat_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e210c00 // sqadd v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_add_sat_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e210c00 // uqadd v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_all_true(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x6e31a801 // uminv b1, v0.16b
WORD $0x0e013c20 // umov w0, v1.b[0]
WORD $0x7100001f // cmp w0, #0
WORD $0x1a9f07e0 // cset w0, ne
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i8x16_avgr_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e211400 // urhadd v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_bitmask(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x4f090401 // sshr v1.16b, v0.16b, #7
MOVD $·simdConstBitpos8(SB), R4
FMOVQ (R4), F2
WORD $0x4e221c21 // and v1.16b, v1.16b, v2.16b
WORD $0x4e083c21 // umov x1, v1.d[0]
WORD $0x4e183c22 // umov x2, v1.d[1]
MOVD $0x0101010101010101, R3
WORD $0x9b037c21 // mul x1, x1, x3
WORD $0x9b037c42 // mul x2, x2, x3
WORD $0xd378fc21 // lsr x1, x1, #56
WORD $0xd378fc42 // lsr x2, x2, #56
WORD $0x2a022020 // orr w0, w1, w2, lsl #8
MOVW R0, ret+16(FP)
RET
TEXT ·simd_i8x16_eq(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e218c00 // cmeq v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_ge_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e213c00 // cmge v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_ge_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e213c00 // cmhs v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_gt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e213400 // cmgt v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_gt_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e213400 // cmhi v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_le_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e203c20 // cmge v0.16b, v1.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_le_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e203c20 // cmhs v0.16b, v1.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_lt_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e203420 // cmgt v0.16b, v1.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_lt_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e203420 // cmhi v0.16b, v1.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_max_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e216400 // smax v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_max_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e216400 // umax v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_min_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e216c00 // smin v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_min_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e216c00 // umin v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_narrow_i16x8_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x0e214800 // sqxtn v0.8b, v0.8h
WORD $0x4e214820 // sqxtn2 v0.16b, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_narrow_i16x8_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x2e212800 // sqxtun v0.8b, v0.8h
WORD $0x6e212820 // sqxtun2 v0.16b, v1.8h
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_ne(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e218c00 // cmeq v0.16b, v0.16b, v1.16b
WORD $0x6e205800 // mvn v0.16b, v0.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_neg(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6e20b800 // neg v0.16b, v0.16b
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i8x16_popcnt(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x4e205800 // cnt v0.16b, v0.16b
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_i8x16_shl(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12000800 // and w0, w0, #7
WORD $0x4e010c01 // dup v1.16b, w0
WORD $0x6e214400 // ushl v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i8x16_shr_s(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12000800 // and w0, w0, #7
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e010c01 // dup v1.16b, w0
WORD $0x4e214400 // sshl v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i8x16_shr_u(SB), NOSPLIT|NOFRAME, $0-40
FMOVQ a+0(FP), F0
MOVWU s+16(FP), R0
WORD $0x12000800 // and w0, w0, #7
WORD $0x4b0003e0 // neg w0, w0
WORD $0x4e010c01 // dup v1.16b, w0
WORD $0x6e214400 // ushl v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+24(FP)
RET
TEXT ·simd_i8x16_shuffle(SB), NOSPLIT|NOFRAME, $0-64
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
FMOVQ pat+32(FP), F2
WORD $0x4e022000 // tbl v0.16b, {v0.16b, v1.16b}, v2.16b
FMOVQ F0, ret+48(FP)
RET
TEXT ·simd_i8x16_splat(SB), NOSPLIT|NOFRAME, $0-24
MOVWU x+0(FP), R0
WORD $0x4e010c00 // dup v0.16b, w0
FMOVQ F0, ret+8(FP)
RET
TEXT ·simd_i8x16_sub(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e218400 // sub v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_sub_sat_s(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e212c00 // sqsub v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_sub_sat_u(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e212c00 // uqsub v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_i8x16_swizzle(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ s+16(FP), F1
WORD $0x4e010000 // tbl v0.16b, {v0.16b}, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_v128_and(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e211c00 // and v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_v128_andnot(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4e611c00 // bic v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_v128_any_true(SB), NOSPLIT|NOFRAME, $0-20
FMOVQ a+0(FP), F0
WORD $0x6e30a801 // umaxv b1, v0.16b
WORD $0x0e013c20 // umov w0, v1.b[0]
WORD $0x7100001f // cmp w0, #0
WORD $0x1a9f07e0 // cset w0, ne
MOVW R0, ret+16(FP)
RET
TEXT ·simd_v128_bitselect(SB), NOSPLIT|NOFRAME, $0-64
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
FMOVQ c+32(FP), F2
WORD $0x4ea21c43 // mov v3.16b, v2.16b
WORD $0x6e611c03 // bsl v3.16b, v0.16b, v1.16b
WORD $0x4ea31c60 // mov v0.16b, v3.16b
FMOVQ F0, ret+48(FP)
RET
TEXT ·simd_v128_not(SB), NOSPLIT|NOFRAME, $0-32
FMOVQ a+0(FP), F0
WORD $0x6e205800 // mvn v0.16b, v0.16b
FMOVQ F0, ret+16(FP)
RET
TEXT ·simd_v128_or(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x4ea11c00 // orr v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
TEXT ·simd_v128_xor(SB), NOSPLIT|NOFRAME, $0-48
FMOVQ a+0(FP), F0
FMOVQ b+16(FP), F1
WORD $0x6e211c00 // eor v0.16b, v0.16b, v1.16b
FMOVQ F0, ret+32(FP)
RET
Reference in New Issue View Git Blame Copy Permalink