mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
Konstantin BogdanovandClaude Fable 5
44f64dd3f0
The x86_64 mem* dispatchers in contrib/libllvmlibc-cmake rely on `flatten`
to inline the whole LLVM-libc helper tree into the `target("arch=x86-64-v4")`
clones. At -O0 (Debug build with -DDEBUG_O_LEVEL=0) clang lowers `flatten`
as a single level of inlining, so the v4-attributed `memmove_overlap_avx512`
calls out-of-line baseline-target (x86-64-v3) helpers that pass 64-byte
vectors by value. That is an ABI mismatch - the v4 caller reads the return
value from zmm0 while the v3 callee returns it in ymm0:ymm1 - which silently
corrupts every overlapping `memmove` of more than 128 bytes on AVX-512
machines. Clang rejects this pattern with a hard error when the mismatched
call is visible in the source, but the inliner recreates it silently.
The corruption made llvm-tblgen crash (SIGSEGV in -gen-x86-fold-tables,
SmallVector `idx < size()` assertion in -gen-asm-matcher) during -O0 builds
on AVX-512 hardware. CI debug builds did not catch it: they run with the
default -Og, and CI machines have no AVX-512.
Fix it in the source so the code is correct at any optimization level:
upgrade LIBC_INLINE to `always_inline` for this TU (honored even at -O0),
and redeclare the generic `load`/`store`/`splat` - which lack LIBC_INLINE
upstream yet are exactly the functions passing vectors by value - with
`always_inline` as well. The helper tree then always collapses into the
exported functions and no cross-target call boundary survives.
Caused by: https://github.com/ClickHouse/ClickHouse/pull/107566
Co-Authored-By: Claude Fable 5 <[email protected]>
61 lines
2.9 KiB
C++
61 lines
2.9 KiB
C++
// Weak `memcmp`, `memcpy`, `memmove`, `memset`, `bcmp` from LLVM-libc.
|
|
//
|
|
// Each exported C symbol is marked weak via LLVM-libc's per-function attribute
|
|
// hook (LLVM_LIBC_FUNCTION_ATTR_<name>), so sanitizer interceptors can override
|
|
// them (the interceptor's strong definition wins the link). As the only
|
|
// non-sanitizer definitions, the weak symbols resolve normally otherwise. The
|
|
// first token after the comma is the attribute placed on the public C symbol;
|
|
// the leading LLVM_LIBC_EMPTY is the sentinel the LLVM-libc macro pipeline
|
|
// consumes.
|
|
|
|
// Upgrade LIBC_INLINE from plain `inline` to `always_inline` for this TU, so
|
|
// the whole LLVM-libc helper tree collapses into the exported functions even
|
|
// at -O0, where the `flatten` attribute on the AVX-512 clones is lowered as a
|
|
// single level of inlining. Without this, the v4-attributed clones call
|
|
// out-of-line baseline-target helpers that pass 64-byte vectors by value - an
|
|
// ABI mismatch (zmm0 vs ymm0:ymm1) that corrupts overlapping `memmove` of
|
|
// more than 128 bytes on AVX-512 machines in -O0 builds.
|
|
#include "src/__support/macros/attributes.h"
|
|
#undef LIBC_INLINE
|
|
#define LIBC_INLINE inline __attribute__((always_inline))
|
|
|
|
// The generic `load`/`store`/`splat` in op_generic.h lack LIBC_INLINE
|
|
// upstream, yet they are exactly the functions that pass vectors by value.
|
|
// Redeclare them with `always_inline` before the definitions are parsed;
|
|
// attributes merge across redeclarations.
|
|
#include "src/string/memory_utils/utils.h"
|
|
|
|
#include <stdint.h>
|
|
|
|
namespace LIBC_NAMESPACE_DECL {
|
|
namespace generic {
|
|
template <typename T> __attribute__((always_inline)) T load(CPtr src);
|
|
template <typename T> __attribute__((always_inline)) void store(Ptr dst, T value);
|
|
template <typename T> __attribute__((always_inline)) T splat(uint8_t value);
|
|
} // namespace generic
|
|
} // namespace LIBC_NAMESPACE_DECL
|
|
|
|
#define LLVM_LIBC_FUNCTION_ATTR_bcmp LLVM_LIBC_EMPTY, __attribute__((weak))
|
|
#define LLVM_LIBC_FUNCTION_ATTR_memcmp LLVM_LIBC_EMPTY, __attribute__((weak))
|
|
#define LLVM_LIBC_FUNCTION_ATTR_memcpy LLVM_LIBC_EMPTY, __attribute__((weak))
|
|
#define LLVM_LIBC_FUNCTION_ATTR_memmove LLVM_LIBC_EMPTY, __attribute__((weak))
|
|
#define LLVM_LIBC_FUNCTION_ATTR_memset LLVM_LIBC_EMPTY, __attribute__((weak))
|
|
|
|
// Include the implementation .cpp files so each picks up the attribute override
|
|
// above before its LLVM_LIBC_FUNCTION expansion.
|
|
// NOLINTBEGIN(bugprone-suspicious-include)
|
|
#include "src/strings/bcmp.cpp" // bcmp lives under <strings.h>, not <string.h>
|
|
#include "src/string/memcmp.cpp"
|
|
#if defined(__x86_64__)
|
|
#include "x86_64_mem_functions.cpp"
|
|
#elif defined(__aarch64__)
|
|
// aarch64 `memcpy`/`memset` come from musl's Arm Optimized Routines assembly;
|
|
// this only adds `memmove`.
|
|
#include "aarch64_mem_functions.cpp"
|
|
#else
|
|
#include "src/string/memcpy.cpp"
|
|
#include "src/string/memmove.cpp"
|
|
#include "src/string/memset.cpp"
|
|
#endif
|
|
// NOLINTEND(bugprone-suspicious-include)
|