Improve performance of __libc_malloc by splitting it into 2 parts: first handle
the tcache fastpath, then do the rest in a separate tailcalled function.
This results in significant performance gains since __libc_malloc doesn't need
to setup a frame and we delay tcache initialization and setting of errno until
later.
On Neoverse V2, bench-malloc-simple improves by 6.7% overall (up to 8.5% for
ST case) and bench-malloc-thread improves by 20.3% for 1 thread and 14.4% for
32 threads.
Reviewed-by: DJ Delorie <dj@redhat.com>
Reject invalid formatted scanf real input data the exponent part of
which is comprised of an exponent introducing character, optionally
followed by a sign, and with no actual digits following. Such data is a
prefix of, but not a matching input sequence and it is required by ISO C
to cause a matching failure.
Currently a matching success is instead incorrectly produced along with
the conversion result according to the input significand read and the
exponent of zero, with the significand and the exponent part wholly
consumed from input.
Correct an invalid `tstscanf.c' test accordingly that expects a matching
success for input data provided in the ISO C standard as an example for
a matching failure.
Enable input data that causes test failures without this fix in place.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Reject invalid formatted scanf real input data that is comprised of a
hexadecimal prefix, optionally preceded by a sign, and with no actual
digits following owing to the field width restriction in effect. Such
data is a prefix of, but not a matching input sequence and it is
required by ISO C to cause a matching failure.
Currently a matching success is instead incorrectly produced along with
the conversion result of zero, with the prefix wholly consumed from
input. Where the end of input is marked by the end-of-file condition
rather than the field width restriction in effect a matching failure is
already correctly produced.
Enable input data that causes test failures without this fix in place.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Reject invalid formatted scanf integer input data that is comprised of a
binary or hexadecimal prefix, optionally preceded by a sign, and with no
actual digits following. Such data is a prefix of, but not a matching
input sequence and it is required by ISO C to cause a matching failure.
Currently a matching success is instead incorrectly produced along with
the conversion result of zero, with the prefix wholly consumed from
input.
Enable input data that causes test failures without this fix in place.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Fix the handling of real 'scanf' input such as "+.e" as per BZ #13988
for the i18n case as well, complementing commit 6ecec3b616 ("Don't
accept exp char without preceding digits in scanf float parsing"), where
the 'e' character is incorrectly consumed from input. Add a test case
matching stdio-common/bug26.c, with bits from localedata/tst-sscanf.c.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add Makefile infrastructure, a format-specific test skeleton providing a
data comparison implementation that ignores bits of data representation
in memory that do not participate in holding floating-point data, and
`long double' real input data for targets using the Intel/Motorola
80-bit format.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0e', '0e+',
'0x', '0x8p', '0x0p-', etc.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
C23 adds various <math.h> function families originally defined in TS
18661-4. Add the pown functions, which are like pow but with an
integer exponent. That exponent has type long long int in C23; it was
intmax_t in TS 18661-4, and as with other interfaces changed after
their initial appearance in the TS, I don't think we need to support
the original version of the interface. The test inputs are based on
the subset of test inputs for pow that use integer exponents that fit
in long long.
As the first such template implementation that saves and restores the
rounding mode internally (to avoid possible issues with directed
rounding and intermediate overflows or underflows in the wrong
rounding mode), support also needed to be added for using
SET_RESTORE_ROUND* in such template function implementations. This
required math-type-macros-float128.h to include <fenv_private.h>, so
it can tell whether SET_RESTORE_ROUNDF128 is defined. In turn, the
include order with <fenv_private.h> included before <math_private.h>
broke loongarch builds, showing up that
sysdeps/loongarch/math_private.h is really a fenv_private.h file
(maybe implemented internally before the consistent split of those
headers in 2018?) and needed to be renamed to fenv_private.h to avoid
errors with duplicate macro definitions if <math_private.h> is
included after <fenv_private.h>.
The underlying implementation uses __ieee754_pow functions (called
more than once in some cases, where the exponent does not fit in the
floating type). I expect a custom implementation for a given format,
that only handles integer exponents but handles larger exponents
directly, could be faster and more accurate in some cases.
I encourage searching for worst cases for ulps error for these
implementations (necessarily non-exhaustively, given the size of the
input space).
Tested for x86_64 and x86, and with build-many-glibcs.py.
Do not build links-dso-program-c with exception (unwinding) support
if libgcc_s is not available. In this case, the unwinder may be
part of libgcc.a or libgcc_eh.a, depending on how GCC was built.
If the unwinder is in libgcc_eh.a only, linking links-dso-program-c
failed before this change. After this change, the exception
handling landing pad is only generated if libgcc_s available,
avoiding an undefined _Unwind_Resume (or equivalent) symbol
reference in the non-libgcc_s case.
Fixes commit ffd36cc27407003a6f9efcb9c16370e3435c5b1d ("support: Use
unwinder in links-dso-program-c only with libgcc_s") and
commit 5dfbc3c43e ("support: Link
links-dso-program-c with libgcc_s only if available").
Use __always_inline for small helper functions that are critical for
performance. This ensures inlining always happens when expected.
Performance of bench-malloc-simple improves by 0.6% on average on
Neoverse V2.
Reviewed-by: DJ Delorie <dj@redhat.com>
Using gcc -Wshift-overflow=2 -Wsystem-headers to compile a file
including <sys/mount.h> will cause a warning since 1 << 31 is undefined
behavior on platforms where int is 32-bits.
Signed-off-by: Collin Funk <collin.funk1@gmail.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
When splitting a chunk, release the tail part by calling int_free_chunk.
This avoids inserting random blocks into tcache that were never requested
by the user. Fragmentation will be worse if they are never used again.
Note if the tail is fairly small, we could avoid splitting it at all.
Also remove an oddly placed initialization of tcache in _libc_realloc.
Reviewed-by: DJ Delorie <dj@redhat.com>
Add Makefile infrastructure and IBM 128-bit 'long double' real input for
targets switching between the IEEE 754 binary128 and IBM 128-bit formats
with '-mabi=ieeelongdouble' and '-mabi=ibmlongdouble'. Reuse IEEE 754
binary128 input data but with modified output file names so as not to
clash with the names used for IBM 128-bit format tests made with common
rules for the 'long double' data type.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0e', '0e+',
'0x', '0x8p', '0x0p-', etc.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add Makefile infrastructure and 64-bit `long double' real input data for
targets switching between the IEEE 754 binary64 and IEEE 754 binary128
formats with `-mlong-double-64' and `-mlong-double-128'. Use modified
output file names for the IEEE 754 binary64 format so as not to clash
with the names used for IEEE 754 binary128 format tests made with common
rules for the 'long double' data type.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0e', '0e+',
'0x', '0x8p', '0x0p-', etc.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add Makefile infrastructure and `long double' real input data for
targets using the IEEE 754 binary128 format.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0e', '0e+',
'0x', '0x8p', '0x0p-', etc.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add Makefile infrastructure and `double' real input data for targets
using the IEEE 754 binary64 format.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0e', '0e+',
'0x', '0x8p', '0x0p-', etc.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add Makefile infrastructure and `float' real input data for targets
using the IEEE 754 binary32 format.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0e', '0e+',
'0x', '0x8p', '0x0p-', etc.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add Makefile infrastructure and `int' and `long' integer input data,
signed and unsigned, for LP64 targets.
While the size of `int' data is the same between ILP32 and LP64 targets,
resulting scanf output is different between them for out of range input
data and while ISO C and POSIX both say that the behavior is undefined
if the result of the conversion cannot be represented we want to keep
track of our output to prevent inadvertent changes. Hence the use of
distinct `int' integer input data between ILP32 and LP64 targets.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0b' or '0x'.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add Makefile infrastructure and `int' and `long' integer input data,
signed and unsigned, for ILP32 targets.
While the size of `int' data is the same between ILP32 and LP64 targets,
resulting scanf output is different between them for out of range input
data and while ISO C and POSIX both say that the behavior is undefined
if the result of the conversion cannot be represented we want to keep
track of our output to prevent inadvertent changes. Hence the use of
distinct `int' integer input data between ILP32 and LP64 targets.
Keep input data disabled and referring to BZ #12701 for entries that are
are currently incorrectly accepted as valid data, such as '0b' or '0x'.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Add a collection of tests for formatted scanf input specifiers covering
the b, d, i, o, u, x, and X integer conversions, the a, A, e, E, f, F,
g, and G floating-point conversions, and the [, c, and s character
conversions. Also the hh, h, l, and ll length modifiers are covered
with the integer conversions as are the l and L length modifier with the
floating-point conversions. The tests cover assignment suppressing and
the field width as well, verifying the number of assignments made, the
number of characters consumed and the value assigned.
Add the common test code here as well as test cases for scanf, and then
base Makefile infrastructure plus target-agnostic input data, for the
character conversions and the `char', `short', and `long long' integer
ones, signed and unsigned, with remaining input data and other functions
from the scanf family deferred to subsequent additions.
Keep input data disabled and referring to BZ #12701 for entries that are
currently incorrectly accepted as valid data, such as '0b' or '0x' with
the relevant integer conversions or sequences of an insufficient number
of characters with the c conversion.
Reviewed-by: Joseph Myers <josmyers@redhat.com>
Check the availability of host_get_time64 and use it to replace
host_get_time for CLOCK_REALTIME when it's available. Fall back to
host_get_time if gnumach does not support host_get_time64 but the
gnumach headers do.
host_get_time is deprecated
See https://git.savannah.gnu.org/cgit/hurd/gnumach.git/commit/?id=569df850cd7badd1e36132ad3b44aa76a4d27c25
However, it's kept for backward compactbility.
* config.h.in: Add HAVE_HOST_GET_TIME64 config entry.
* sysdeps/mach/clock_gettime.c: Use host_get_time64 for CLOCK_REALTIME
when it's possible, fall to host_get_time otherwise.
* sysdeps/mach/configure: Check the existence of host_get_time64 RPC.
* sysdeps/mach/configure.ac: Check the existence of host_get_time64 RPC.
Message-ID: <20250324052042.19803-1-zhmingluo@163.com>
cc5d5852c6 ("y2038: Convert aio_suspend to support 64 bit time")
switched from __clock_gettime (CLOCK_REALTIME, &now); to __clock_gettime64
(CLOCK_MONOTONIC, &ts);, but pthread_cond_timedwait is based on the
absolute realtime clock, so migrate to using pthread_cond_clockwait to
select CLOCK_MONOTONIC. Also fix AIO_MISC_WAIT into passing
CLOCK_MONOTONIC to __futex_abstimed_wait64.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Add function __inet_pton_chk which calls __chk_fail when the size of
argument dst is too small. inet_pton is redirected to __inet_pton_chk
or __inet_pton_warn when _FORTIFY_SOURCE is > 0.
Also add tests to debug/tst-fortify.c, update the abilist with
__inet_pton_chk and mention inet_pton fortification in maint.texi.
Co-authored-by: Frédéric Bérat <fberat@redhat.com>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Split inet_pton internals such as __inet_pton_length from the
inet_pton entry point.
This allows the internals to be built with fortification while
leaving the inet_pton entry point unchanged.
Co-authored-by: Frédéric Bérat <fberat@redhat.com>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
There are no new constants covered by tst-mman-consts.py,
tst-mount-consts.py or tst-sched-consts.py in Linux 6.13 that need any
header changes, so update the kernel version in those tests.
(tst-pidfd-consts.py will need updating separately along with adding
new constants to glibc.)
Tested with build-many-glibcs.py.
Add a configure check to detect bootstrapping builds that do not
have libgcc_s.
Fixes commit 3e2be87832 ("support: Link
links-dso-program-c against libgcc_s").
Reviewed-by: Sam James <sam@gentoo.org>
The tst-origin test must link against liborigin-mod.so. Correct
build order depends on a makefile rule dependency on
$(objpfx)liborigin-mod.so. Use +nolink-deps to remvoe this
dependency from the linker command line.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
This allows adding dependencies to rules, but not linking against
them.
This is more or less a reimplementation of .EXTRA_PREREQS feature from
GNU make 4.3.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The 7bb8045ec0 path made the '%n' fortify check ignore EMFILE errors
while trying to open /proc/self/maps, and this added a security
issue where EMFILE can be attacker-controlled thus making it
ineffective for some cases.
The EMFILE failure is reinstated but with a different error
message. Also, to improve the false positive of the hardening for
the cases where no new files can be opened, the
_dl_readonly_area now uses _dl_find_object to check if the
memory area is within a writable ELF segment. The procfs method is
still used as fallback.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Arjun Shankar <arjun@redhat.com>
On both Linux and Hurd the __eloop_threshold() is always a constant
(40 and 32 respectively), so there is no need to always call
__sysconf (_SC_SYMLOOP_MAX) for Linux case (!SYMLOOP_MAX). To avoid
a name clash with gnulib, rename the new file min-eloop-threshold.h.
Checked on x86_64-linux-gnu and with a build for x86_64-gnu.
Reviewed-by: DJ Delorie <dj@redhat.com>
If C++ support is not available, links-dso-program-c is used
instead of the C++ version. The C version was not linked against
libgcc_s, which meant that thread cancellation and the backtrace
function did not work in containers tests in that situation.
Reviewed-by: Sam James <sam@gentoo.org>
- Create the __inet_ntop_chk routine that verifies that the builtin size
of the destination buffer is at least as big as the size given by the
user.
- Redirect calls from inet_ntop to __inet_ntop_chk or __inet_ntop_warn
- Update the abilist for this new routine
- Update the manual to mention the new fortification
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Rename inet_ntop to __inet_ntop and create the inet_ntop weak alias
based on it in order to prepare for disabling fortification when
available.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Remove the alignment rounding up from csize2tidx - this makes no sense
since the input should be a chunk size. Removing it enables further
optimizations, for example chunksize_nomask can be safely used and
invalid sizes < MINSIZE are not mapped to a valid tidx.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The tst-origin build can fail with:
/usr/bin/ld: [...]libc.so: undefined reference to `__tunable_is_initialized@GLIBC_PRIVATE'
Since the custom link invocation links against system glibc instead
of the built one.
The only requirement is to avoid liborigin.so linked with a full path,
which is the default for --enable-hardcoded-path-in-tests. There
is no need to use a custom rule.
Checked on x86_64-linux-gnu.
Polynomial order was unnecessarily high, unlocking multiple
optimizations.
Max error for new SVE expf is 0.88 +0.5ULP.
Max error for new SVE coshf is 2.56 +0.5ULP.
Performance improvement on Neoverse V1: expf (30%), coshf (26%).
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Change heap_max_size() to improve performance of arena_for_chunk().
Instead of a complex calculation, using a simple mask operation to get the
arena base pointer. HEAP_MAX_SIZE should be larger than the huge page size,
otherwise heaps will use not huge pages.
On AArch64 this removes 6 instructions from arena_for_chunk(), and
bench-malloc-thread improves by 1.1% - 1.8%.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The fread race checker looks for EOF in every thread, which is incorrect
since threads calling fread successfully could lag behind and read the
EOF condition, resulting in multiple threads thinking that they
encountered an EOF.
Only look for EOF condition if fread fails to read a char. Also drop
the clearerr() since it could mask the failure of another reader, thus
hiding a test failure.
Finally, also check for error in the stream for completeness.
Signed-off-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
C23 adds various <math.h> function families originally defined in TS
18661-4. Add the powr functions, which are like pow, but with simpler
handling of special cases (based on exp(y*log(x)), so negative x and
0^0 are domain errors, powers of -0 are always +0 or +Inf never -0 or
-Inf, and 1^+-Inf and Inf^0 are also domain errors, while NaN^0 and
1^NaN are NaN). The test inputs are taken from those for pow, with
appropriate adjustments (including removing all tests that would be
domain errors from those in auto-libm-test-in and adding some more
such tests in libm-test-powr.inc).
The underlying implementation uses __ieee754_pow functions after
dealing with all special cases that need to be handled differently.
It might be a little faster (avoiding a wrapper and redundant checks
for special cases) to have an underlying implementation built
separately for both pow and powr with compile-time conditionals for
special-case handling, but I expect the benefit of that would be
limited given that both functions will end up needing to use the same
logic for computing pow outside of special cases.
My understanding is that powr(negative, qNaN) should raise "invalid":
that the rule on "invalid" for an argument outside the domain of the
function takes precedence over a quiet NaN argument producing a quiet
NaN result with no exceptions raised (for rootn it's explicit that the
0th root of qNaN raises "invalid"). I've raised this on the WG14
reflector to confirm the intent.
Tested for x86_64 and x86, and with build-many-glibcs.py.
Wilco Dijkstra
Maciej W. Rozycki
Joseph Myers
Florian Weimer
Collin Funk
Zhaoming Luo
Samuel Thibault
Aaron Merey
Adhemerval Zanella
Cupertino Miranda
Frédéric Bérat
Paul Zimmermann
Pierre Blanchard
Siddhesh Poyarekar