Increase iterations for strpbrk/strspn so they run for at least 0.5s.
Reduce iterations for the much slower wcs* variants since they take over 10s.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Remove the glibc.cpu.name tunable since it's unused and out of date.
Add support for glibc.cpu.hwcaps to adjust ifunc selection for debugging
and benchmarking. Only allow disabling of features that are (a) used by
ifuncs, (b) safe to disable to a more generic ifunc without any security
impact.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
Add matches before the start to ensure the memchr test checks they do not
result in false matches or missed matches.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Update memchr description to indicate it cannot overread and the size
may be larger than the object passed. This is required since C11.
Suggested-by: Paul Eggert <eggert@cs.ucla.edu>
Reviewed-by: Paul Eggert <eggert@cs.ucla.edu>
Simplify _int_free_chunk() and always lock if needed. Use
_int_free_merge_chunk() for cases that assume the arena has been locked
instead. Move the errno save/restore to _int_free_maybe_trim().
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
The default maximum arenas is 8 times the number of cores in a 64-bit system.
Since modern CPUs have many cores and big servers have 256 cores, this results
in excessive number of arenas, which wastes memory. Limit the number of arenas
to max (8, ncores) which is less extreme. In the future the limit should be
lowered further for large systems.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Update default for tcache_count tunable. Remove existing documentation and
mention removal of fastbins in mxfast tunable. Improve wording of hugetlb
tunable, including default for AArch64.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Merge request2size into checked_request2size. Improve interface of
clear_memory. Fix incorrect use of __glibc_unlikely in __libc_calloc.
Fix missing tabs.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
The tests for memchr attempt to check for accidental overreads that cross a
page. However they weren't done at the end of a page and don't check for the
case where we match right at the end. Add buf1_size/buf2_size to make finding
end of buffer easier.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
v2: Update documentation
Whenever a large mmap is released the mmap and trim thresholds are updated.
As a result these thresholds grow ever larger which means huge allocations
are always served by arenas rather than mmap. The thresholds can end up as
large as an arena, which completely stops all trimming of the top block.
Remove the code completely - the default thresholds seem way too low for
modern 64-bit targets, but they can be increased seperately.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Malloc defines a lot of aliases which are unused, not exported and not needed.
Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: DJ Delorie <dj@redhat.com>
Update the hugetlb tunable default in elf/dl-tunables.c so it is shown as 1
with /lib/ld-linux-aarch64.so.1 --list-tunables.
Move the intitialization of thp_mode/thp_pagesize to do_set_hugetlb() and
avoid accessing /sys/kernel/mm if DEFAULT_THP_PAGESIZE > 0. Switch off THP if
glibc.malloc.hugetlb=0 is used - this behaves as if DEFAULT_THP_PAGESIZE==0.
Fix the --list-tunables testcase.
Reviewed-by: DJ Delorie <dj@redhat.com>
Remove the prefer_sve_ifuncs CPU feature since it was intended for older
kernels. Current distros all use modern Linux kernels with improved support
for SVE save/restore, making this check redundant.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
Cleanup warnings - malloc builds with -Os and -Og without needing any
complex warning avoidance defines.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Use generic stdc_bit_width to safely adapt to input types. Move rounding up of
alignments that are not powers of 2 to __libc_memalign. Simplify alignment
handling of aligned_alloc and __posix_memalign. Add a testcase for non-power
of 2 memalign and fix malloc-debug.
Reviewed-by: DJ Delorie <dj@redhat.com>
On AArch64 malloc always checks /sys/kernel/mm/transparent_hugepage/enabled to
set the THP mode. However this check is quite expensive and the file may not
be accessible in containers. If DEFAULT_THP_PAGESIZE is non-zero, use
malloc_thp_mode_madvise so that we take advantage of THP in all cases. Since
madvise is a fast systemcall, it adds only a small overhead compared to the
cost of mmap and populating the pages.
Reviewed-by: Aurelien Jarno <aurelien@aurel32.net>
Change the mask to 48 to support len==64. The second memory store now accesses
offset 32, whereas the third one accesses offset 16. As a result performance
for len==64 almost doubles.
Currently malloc has various assumptions, some documented, some implicit.
Add a few asserts to check the most fundamental assumptions using verify().
Remove some odd #define void.
Reviewed-by: Paul Eggert <eggert@cs.ucla.edu>
Now that fastbins have been removed, there is no need to add chunks
to tcache during an unsorted scan. Small blocks can only be added
to unsorted as a result of a remainder chunk split off a larger block,
so there is no point in checking for additional chunks to place in
tcache. The last remainder is checked first, and will be used if it
is large enough or an exact fit. The unsorted bin scan becomes simpler
as a result. Remove the tcache_unsorted_limit tunable and manual entries.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Only enable vector math functions with -ffast-math by adding if('fastmath').
This uses a new annotation supported by GCC 16 (PR 118955). If an older
compiler is used, it will disable the math function without an error.
This fixes BZ #33226.
Add __issignaling inline based on the issignaling_inline implementation.
Improve the __issignalingf inline. Remove issignaling(f)_inline and its uses.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Co-authored-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reinstate HAVE_64B_ATOMICS configure check that was reverted by commit
7fec8a5de6 due to BZ #33632. This was
fixed by 3dd2cbfa35 by only allowing
64-bit atomics on sem_t if its type is 8-byte aligned. Rebase and add
in cleanups in include/atomic.h that were omitted.
Fix an issue with sparcv8-linux-gnu-leon3 forcing -mcpu=v8 for rtld.c which
overrules -mcpu=leon3 and causes __atomic_always_lock_free (4, 0) to
incorrectly return 0 and trigger asserts in atomics. Remove this as it
seems to be a workaround for an issue in 1997.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Cleanup thp_init, change it so that the DEFAULT_THP_PAGESIZE
setting can be overridden with glibc.malloc.hugetlb=0 tunable.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Report assertion failure if the alignment of external pthread structs is
lower than the internal version. This triggers on type mismatches like
in BZ #33632.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
Directly call _int_free_chunk during tcache shutdown to avoid recursion.
Calling __libc_free on a block from tcache gets flagged as a double free,
and tcache_double_free_verify checks every tcache chunk (quadratic
overhead).
Reviewed-by: Arjun Shankar <arjun@redhat.com>
Remove uses of float_t and double_t. This is not useful on modern machines,
and does not help given GCC defaults to -fexcess-precision=fast.
One use of double_t remains to allow forcing the precision to double
on targets where FLT_EVAL_METHOD=2. This fixes BZ #33563 on
i486-pc-linux-gnu.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Remove ldbl-128/s_fma.c - it makes no sense to use emulated float128
operations to emulate FMA. Benchmarking shows dbl-64/s_fma.c is about
twice as fast. Remove redundant dbl-64/s_fma.c includes in targets
that were trying to work around this issue.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The only usage was for pthread_spin_lock, introduced by 12d2dd7060,
as a way to optimize the code for certain architectures. Now that atomic
builtins are used by default, let the compiler use the best code sequence
for the atomic exchange.
Co-authored-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Now that atomic builtins are used by default, we can rely on the
compiler to define when to use 64-bit atomic operations.
It allows the use of 64-bit atomic operations on some 32-bit ABIs where
they were not previously enabled due to missing pre-processor handling:
hppa, mips64n32, s390, and sparcv9.
Co-authored-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Reviewed-by: Uros Bizjak <ubizjak@gmail.com>
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Remove xfail from pow testcase since pow and powf have been fixed.
Also check float128 maximum value. See BZ #33563.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Fix pow (DBL_MAX, 1.0) to return DBL_MAX when rouding upwards without FMA.
This fixes BZ #33563.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Fix powf (0x1.fffffep+127, 1.0f) to return 0x1.fffffep+127 when
rouding upwards. Cleanup the special case code - performance
improves by ~1.2%. This fixes BZ #33563.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Remove target implementations of roundeven(f)/lrint(f)/lround(f) and
use the math-use-builtins mechanism instead.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Cleanup _int_memalign. Simplify the logic. Add a seperate check
for mmap. Only release the tail chunk if it is at least MINSIZE.
Use the new mmap abstractions.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Add mmap_set_chunk() to create a new chunk from an mmap block.
Remove set_mmap_is_hp() since it is done inside mmap_set_chunk().
Rename prev_size_mmap() to mmap_base_offset(). Cleanup comments.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Remove the odd atomic_forced_read which is neither atomic nor forced.
Some uses are completely redundant, so simply remove them. In other cases
the intended use is to force a memory ordering, so use acquire load for those.
In yet other cases their purpose is unclear, for example __nscd_cache_search
appears to allow concurrent accesses to the cache while it is being garbage
collected by another thread! Use relaxed atomic loads here to block spills
from accidentally reloading memory that is being changed.
Passes regress on AArch64, OK for commit?
Remove support for obsolete dumped heaps. Dumping heaps was discontinued
8 years ago, however loading a dumped heap is still supported. This blocks
changes and improvements of the malloc data structures - hence it is time
to remove this. Ancient binaries that still call malloc_set_state will now
get the -1 error code. Update tst-mallocstate.c to just check for this.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Minor cleanup of libc_realloc: remove unnecessary special cases for mmap, move
ar_ptr initialization, first check for oldmem == NULL.
Reviewed-by: DJ Delorie <dj@redhat.com>
Remove all unused atomics. Replace uses of catomic_increment and
catomic_decrement with atomic_fetch_add_relaxed which maps to a standard
compiler builtin. Relaxed memory ordering is correct for simple counters
since they only need atomicity.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Switch to standard builtin atomics by removing the defines for
!USE_ATOMIC_COMPILER_BUILTINS.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Use builtin atomics for atomic_compare_and_exchange_* and
atomic_exchange_and_add if USE_ATOMIC_COMPILER_BUILTINS is enabled.
This allows removing target atomic-machine.h headers.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Add mremap_chunk support for mmap()ed chunks using hugepages by accounting for
their alignment, to prevent the mremap call failing in most cases where the
size passed is not a hugepage size multiple. It also improves robustness for
reallocating hugepages since mremap is much less likely to fail, so running
out of memory when reallocating a larger size and having to copy the old
contents after mremap fails is also less likely.
To track whether an mmap()ed chunk uses hugepages, have a flag in the lowest
bit of the mchunk_prev_size field which is set after a call to sysmalloc_mmap,
and accessed later in mremap_chunk. Create macros for getting and setting this
bit, and for mapping the bit off when accessing the field for mmap()ed chunks.
Since the alignment cannot be lower than 8 bytes, this flag cannot affect the
alignment data.
Add malloc/tst-tcfree4-malloc-check to the tests-exclude-malloc-check list as
malloc-check prevents the tcache from being used to store chunks. This test
caused failures due to a bug in mem2chunk_check to be fixed in a later patch.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Change the mmap chunk layout to be identical to a normal chunk. This makes it
safe for tcache to hold mmap chunks and simplifies size calculations in
memsize and musable. Add mmap_base() and mmap_size() macros to simplify code.
Reviewed-by: Cupertino Miranda <cupertino.miranda@oracle.com>
MALLOC_DEBUG only works on locked arenas, so move the call to
check_inuse_chunk from __libc_free() to _int_free_chunk().
Regress now passes if MALLOC_DEBUG is enabled.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Arenas support huge pages but not transparent huge pages. Add this by
also checking mp_.thp_pagesize when creating a new arena, and use madvise.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Remove an odd use of __curbrk and use MORECORE (0) instead.
This fixes Hurd build since it doesn't define this symbol.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Cleanup sysmalloc_mmap - simplify padding since it is always a constant.
Remove av parameter which is only used in do_check_chunk, but since it may be
NULL for mmap, it will cause a crash in checking mode. Remove the odd check
on mmap in do_check_chunk.
Reviewed-by: DJ Delorie <dj@redhat.com>
Change checked_request2size to return SIZE_MAX for huge inputs. This
ensures large allocation requests stay large and can't be confused with a
small allocation. As a result several existing checks against PTRDIFF_MAX
become redundant.
Reviewed-by: DJ Delorie <dj@redhat.com>
MAX_TCACHE_SMALL_SIZE should use chunk size since it is used after
checked_request2size. Increase limit of tcache_max_bytes by 1 since all
comparisons use '<'. As a result, the last tcache entry is now used as
expected.
Reviewed-by: DJ Delorie <dj@redhat.com>
During early startup memcpy or memset must not be called since many targets
use ifuncs for them which won't be initialized yet. Security hardening may
use -ftrivial-auto-var-init=zero which inserts calls to memset. Redirect
memset to memset_generic by including dl-symbol-redir-ifunc.h in cpu-features.c.
This fixes BZ #33112.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Remove unused 'address' parameter from _mid_memalign and callers.
Fix off-by-one alignment calculation in __libc_pvalloc.
Reviewed-by: DJ Delorie <dj@redhat.com>
Currently tcache requires 2 global variable accesses to determine
whether a block can be added to the tcache. Change the counts array
to 'num_slots' to indicate the number of entries that could be added.
If 'num_slots' reaches zero, no more blocks can be added. If the entries
pointer is not NULL, at least one block is available for allocation.
Now each tcache bin can support a different maximum number of entries,
and they can be individually switched on or off (a zero initialized
num_slots+entry means the tcache bin is not available for free or malloc).
Reviewed-by: DJ Delorie <dj@redhat.com>
When using a -mcpu option in CFLAGS, GCC can report errors when building libmvec.
Fix this by overriding both -mcpu and -march with a generic variant with SVE added.
Also use a tune for a modern SVE core.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
__ptmalloc_init was called too early in __libc_early_init: it uses
__libc_initial which is not set yet. Fix this by moving initialization
to the end of __libc_early_init.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Add initial inputs for asinpi(f), acospi(f), atanpi(f) and atan2pi(f) based
on existing asin/acos/atan inputs.
Benchtests now works on the new libmvec function.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
Now we finally support modern GCC and binutils, it's time for a cleanup.
Remove HAVE_AARCH64_SVE_ASM define and conditional compilation. Remove SVE
configure checks for SVE, ACLE and variant-PCS support.
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
Now we finally support modern GCC and binutils, it's time for a cleanup.
Use PAC and BTI instructions unconditionally and use proper assembler syntax.
Remove the PR target/94791 strip_pac workarounds for buggy GCCs. Remove the
PAC/BTI configure checks - always emit GNU property notes on assembly files.
Change cfi_window_save to the correct cfi_negate_ra_state unwind directive.
Reviewed-by: Matthieu Longo <matthieu.longo@arm.com>
Improve performance of __libc_calloc by splitting it into 2 parts: first handle
the tcache fastpath, then do the rest in a separate tailcalled function.
This results in significant performance gains since __libc_calloc doesn't need
to setup a frame.
On Neoverse V2, bench-calloc-simple improves by 5.0% overall.
Bench-calloc-thread 1 improves by 24%.
Reviewed-by: DJ Delorie <dj@redhat.com>
Move malloc initialization to __libc_early_init. Use a hidden __ptmalloc_init
for initialization and a weak call to avoid pulling in the system malloc in a
static binary. All previous initialization checks can now be removed.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Inline tcache_try_malloc into calloc since it is the only caller. Also fix
usize2tidx and use it in __libc_malloc, __libc_calloc and _mid_memalign.
The result is simpler, cleaner code.
Reviewed-by: DJ Delorie <dj@redhat.com>
Use tailcalls to avoid the overhead of a frame on the free fastpath.
Move tcache initialization to _int_free_chunk(). Add malloc_printerr_tail()
which can be tailcalled without forcing a frame like no-return functions.
Change tcache_double_free_verify() to retry via __libc_free() after clearing
the key.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Wilco Dijkstra
