For AT_SECURE programs the loader honors $ORIGIN in DT_RPATH only when the
expansion is rooted in a trusted directory, but it validated the lexically
normalized path while opening the raw expansion. As "a/b/../c" only names
"a/c" when "b" is not a symlink, an attacker who controls a component of
$ORIGIN -- e.g. by hard-linking the setuid binary into an attacker-owned
directory -- can make the opened path escape the trusted directory even
though the check passed, loading an attacker-controlled object.
Normalize the expansion in place and open that, so the path that is opened
is exactly the path that was validated. _dl_normalize_path rewrites the
string in place without ever advancing its write cursor past its read
cursor or appending, so it stays within the original storage.
Add elf/tst-origin-secure as a regression test.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Since af34b1376a ("elf: Initialize static TLS before relocation
processing", BZ 34164) dropped the 'defer-if-not-relocated' branch in
_dl_try_allocate_static_tls, nothing sets l_need_tls_init any more. The
second pass in update_tls_slotinfo, guarded by l_need_tls_init, is
therefore dead: its _dl_update_slotinfo / _dl_init_static_tls calls never
run, and the static TLS image is initialised inline during relocation (IE
model) or lazily on first dynamic-TLS access instead.
Remove the dead loop, the now write-only l_need_tls_init field and its
clear in _dl_allocate_tls_init. No functional change.
Checked on aarch64-linux-gnu, x86_64-linux-gnu, and i686-linux-gnu.
I also run the elf tests on armv7-a, alpha, loongarch64, mips64le,
powerpc, riscv, and s390x using qemu system.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
The initial static-pie support (commit 9d7a3741c9) reused
ld.so ELF parsing logic, even though RPATH/RUNPATH should not appear
in the static-pie bootstrap. With static PIE, RPATH/RUNPATH on the
loader typically indicates a toolchain misconfiguration. However,
for static PIE, the presence of RPATH/RUNPATH has no impact because
these binaries do not use dynamic linking at runtime.
Fully static binaries have no dynamic section, so RPATH/RUNPATH can not
appear there at all; for static PIE the only consumer is dlopen. If
static dlopen support is ever removed, this change becomes a no-op.
This change also simplifies elf_get_dynamic_info and removes a
difference between dynamic and static binaries, along with the now
unused STATIC_PIE_BOOTSTRAP.
Tested on aarch64-linux-gnu and x86_64-linux-gnu.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
The BZ#33326 testcase triggers an assertion during process startup,
which results in a segmentation fault instead of an error message
and process termination with a SIGABRT. The assert issues
__libc_message_impl, which in turn might call string functions
depending on the ABI (strchrnul, strlen, memcpy/mempcpy), system
calls (writev and mmap), and finally the abort call.
The dl-symbol-redir-ifunc.h is also expanded to cover strchrnul on
x86_64, s390, powerpc64 (both endianness) and loongarch, mempcpy on
powerpc64be, and memcpy on aarch64. On s390 the redirection is only
issued if the ifunc variant is built, since strchrnul-c.c only renames
the C implementation to STRCHRNUL_DEFAULT when HAVE_STRCHRNUL_IFUNC is
set.
The buffer that backs up the assert message is now allocated through
_dl_mmap, which issues the syscall directly instead of calling __mmap
(setting errno on failure requires the thread pointer).
The abort call now issues __raise_direct instead of raise (the Hurd
port aliases __raise_direct to raise).
On i386, syscalls should not use the vDSO during program startup because
the thread pointer is not yet initialized. This requires __raise_direct,
_dl_writev, and _dl_mmap to be built with I386_USE_SYSENTER set to 0.
Creating a test case is challenging. For static-pie, the assert is only
called for ill-formed ELF files on elf_get_dynamic_info and by some targets
on ELF_DYNAMIC_RELOCATE (although not all targets use assert in their
dl-machine.h). Some targets also issue __libc_fatal on ARCH_SETUP_IREL,
but also only for ill-formatted ELF files.
The test employs a different strategy and overrides the __tunables_init
symbol, which is invoked immediately before self-relocation and TLS setup.
The test is built with -Wl,-z,muldefs to avoid linker issues.
I checked on aarch64, x86_64, i686, s390x (qemu), sparc (qemu),
mips64el (qemu), armhf, riscv, and powerpc.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
And change _dl_writev to return a negative errno in case of failure.
This keeps the required semantics for not setting errno on failure
and allows removing the Linux libc_fatal.c implementation.
It also makes it simple to use the writev syscall during process
startup, especially on i386, where it requires disabling vDSO.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
commit 6deadd4eb6
Author: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Date: Wed Oct 8 10:55:05 2025 -0300
didn't remove sysdeps/m68k/m680x0/fpu/w_fmod_compat.c. As the result,
due to a linker bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=34550
there were 2 default versions of fmod in m68k libm:
996: 0001433c 174 FUNC WEAK DEFAULT 12 fmod@@GLIBC_2.0
997: 000307d4 214 FUNC GLOBAL DEFAULT 12 fmod@@GLIBC_2.43
Add check-symbol-version.awk to verify that versioned symbols only have
one default version in dynamic symbol table.
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Sam James <sam@gentoo.org>
A string tunable value usually references the GLIBC_TUNABLES (or alias)
environment string, which lives in the environment block the kernel places
on the initial stack. That memory is owned by the application, which may
overwrite it (e.g. setproctitle), so the reference is only safe while no
application code has run (a value coming from the system-wide tunables
cache is a copy instead, but the rule is applied uniformly).
This patch make the lifetime explicit and enforced without copying the value
or allocating any memory by adding __tunable_seal_strings, which drops every
string tunable reference once early startup is complete.
The seal is applied after the only string tunable consumer and before any
code outside of the startup sequence runs.
Checked on aarch64-linux-gnu and x86_64-linux-gnu. I also run the elf
tests on powerpc64le-linux-gnu, loongarch64-linux-gnuf64, and
s390x-linux-gnu.
_dl_sort_maps is only called during shared object loading and
unloading, not on any hot path. The branch predictor hint has no
measurable benefit, and removing it simplifies the code and avoids
future maintenance when the default algorithm changes.
Suggested-by: Wilco Dijkstra <wilco.dijkstra@arm.com>
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
The three resolver translation units are compiled with
-fstack-protector-all so that the canary code is emitted whatever the
default is. A target whose compiler has no stack protector at all cannot
do that, and on alpha the build stopped:
cc1: error: '-fstack-protector' not supported for this target [-Werror]
taking the rest of the elf tests with it. Guard the tests on $(have-ssp),
which configure already sets from the compiler's own answer. That is the
answer for -fstack-protector rather than for -fstack-protector-all, which
has its own configure test, but only the former reaches config.make, and a
compiler that has one has the other.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
clang-23 warns that the variable 'ext' set but not used:
dl-cache.c:42:47: error: variable 'ext' set but not used
[-Werror,-Wunused-but-set-global]
The variable is set unconditionally by _dl_load_cache_lookup, but it
is only read by glibc_hwcaps_priorities_init (which is compiled only
for SHARED).
clang enables -Wsingle-bit-bitfield-constant-conversion with -Werror
and it triggers:
tunconf.c:338:32: error: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Werror,-Wsingle-bit-bitfield-constant-conversion]
338 | entry->value_is_negative = 1;
Change both value_is_negative and value_was_parsed to unsigned.
Checked on x86_64-linux-gnu and i686-linux-gnu.
The 63b31c05a8 split relocation processing must agree for the lazy
flag, a mismatch would change the .rel.plt handling.
This is not an issue for any port currently, but on hppa it may return
a different value: if hppa implements IFUNC support, the second pass would
route PLT entries to its empty lazy handler and leave the descriptors
unrelocated, silently.
Make ELF_DYNAMIC_RELOCATE_PASS take lazy as an int lvalue and store the
effective mode back into it, so the DL_RELOC_IRELATIVE call reuses the
same variable instead of a separately threaded copy. The two passes can no
longer disagree about the partitioning. elf_machine_runtime_setup has side
effects, so it must stay a single call.
Checked on x86_64-linux-gnu, and built for all supported architectures.
Reviewed-by: Sam James <sam@gentoo.org>
Commit 63b31c05a8 ("elf: Defer all IRELATIVE relocations until after PLT
setup") dropped the skip_ifunc argument from elf_dynamic_do_Rel, assuming
the new deferred elf_dynamic_do_Rel_irelative pass handles every relocation
that may run an IFUNC resolver. That only holds for IFUNC symbols defined
in the object being relocated: a reference to an IFUNC in another object is
an ordinary JMP_SLOT or GLOB_DAT against an undefined symbol, and its IFUNC
nature is only known after symbol resolution inside elf_machine_rel. Those
relocations stay in the regular pass, which no longer propagated
skip_ifunc, so __RTLD_NOIFUNC was ignored for them.
ldd -u forces non-lazy binding (GLRO(dl_lazy) = 0 for DL_DEBUG_UNUSED), so
the resolver was called and the diagnostic emitted:
$ ldd -u /bin/ls
/bin/ls: Relink `' with `/usr/lib64/libc.so.6' for IFUNC symbol `__mempcpy_chk'
ldd -r with LD_BIND_NOW is affected in the same way.
Restore the skip_ifunc parameter and thread it through _ELF_DYNAMIC_DO_RELOC.
This new semantic shows that ELF_DYNAMIC_RELOCATE_NOIFUNC naming is misleading
(it reads as "do not process IFUNC", yet it takes a skip_ifunc
argument). Replace it to:
DL_RELOC_BOTH -> DL_RELOC_ALL
DL_RELOC_NOIFUNC -> DL_RELOC_NORMAL
DL_RELOC_IFUNC -> DL_RELOC_IRELATIVE
ELF_DYNAMIC_RELOCATE_NOIFUNC and ELF_DYNAMIC_RELOCATE_IFUNC become a single
ELF_DYNAMIC_RELOCATE_PASS taking the pass as its first argument, and
ELF_DYNAMIC_DO_REL/ELF_DYNAMIC_DO_RELA take the pass instead of having three
near-identical variants each.
Checked on x86_64-linux-gnu, and built for all supported architectures.
Reviewed-by: Sam James <sam@gentoo.org>
Use $fail when we set it rather than just '1' (we already checked
that it is non-zero) to make logs more meaningful.
(Also, use $rc when we already checked it svalue.)
The cancellable syscall wrappers end with a tail call to __syscall_cancel,
the wrapper frame is then elided, so when the syscall executes the wrapper
is no longer present on the stack. Tools that unwind from CFI alone, such
as valgrind, perf and sampling profilers, cannot observe it. On gdb, it
only recovers it from DWARF call site information, which reduced-debuginfo
libc builds usually omit.
The behaviour is target dependent: for a shared (PIC) the tail call is
emitted on aarch64, arc, loongarch and riscv. It is not emitted on i386,
x86_64, arm, s390x, sparc and alpha, where the seventh argument is passed
on the stack or fewer argument registers are available, nor on powerpc
and mips, where the TOC/GOT pointer must be restored after the call.
This is why the problem was originally reported as aarch64 specific while
x86_64 was unaffected.
Rather than only inhibiting the tail call [1] (which keeps the wrapper frame
but still leaves the __syscall_cancel and __internal_syscall_cancel
frames), move the cancellation logic back into the wrappers. In the
single-threaded case the syscall is now issued directly from the wrapper;
only the multi-threaded path still calls the out-of-line __syscall_cancel_arch.
This keeps the wrapper observable and removes the extra frames, mimicking
how cancellation was handled before 89b53077d2.
The result is a small libc.so .text increase (size, first column):
ABI master patched diff increase
aarch64 1635880 1647424 11544 0.71%
x86_64 1981081 1992257 11176 0.56%
powerpc64le 2364336 2376964 12628 0.53%
riscv64 1368386 1376704 8318 0.61%
loongarch64 1741385 1755601 14216 0.82%
The tst-backtrace5 was suppose to track this issue, but due wrong
loop variable check it does not take this in account. This patch also fixes
it.
Checked on aarch64-linux-gnu, x86_64-linux-gnu, i686-linux-gnu,
arm-linux-gnueabihf, and powerpc64le-linux-gnu.
[1] https://sourceware.org/pipermail/libc-alpha/2025-March/165395.html
Improve the error diagnostics printed when static TLS allocation fails
during dlopen.
The CHECK_STATIC_TLS macro is updated to pass the fully resolved sym and
the referencing map over to _dl_allocate_static_tls, modifying its
signature.
When _dl_allocate_static_tls is called, it now attempts to reconstruct
what failed using _dl_exception_create_format. It displays:
* The name of the symbol that triggers this.
* Whether this is due to static TLS space being exhausted, or if the
symbol has previously been used as global-dynamic and is now being
tried to use as initial-exec.
* If the symbol-defining map is different from the referencing map, it
includes its name as well.
* If static TLS is exhausted, includes requested size and available
size.
The change cascades through all architecture variants modifying their
dl-machine calls to CHECK_STATIC_TLS to conform to the new prototype.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
When compiling a glibc for a merged-/usr distro people may set
rootsbindir=/usr/sbin. But tst-ldconfig-cache has hard-coded
/sbin/ldconfig path and so it fails with a different rootsbindir.
Fix it by using support_install_rootsbindir like run_ldconfig in
test-container.c.
Signed-off-by: Xi Ruoyao <xry111@xry111.site>
Reviewed-by: Florian Weimer <fweimer@redhat.com>
The split introduced by commit 63b31c05a8 does not handle sparc
and (R_SPARC_JMP_IREL) powerpc64 (ELFv1, R_PPC64_JMP_IREL), which
are emited in some constructions. Handle such cases on
elf_dynamic_is_Rel_irelative.
It fixes elf/tst-ifunc-fault-bindnow and elf/tst-ifunc-fault-lazy on
sparc64 (powerpc64 emits R_PPC64_IRELATIVE in both cases,
R_PPC64_JMP_IREL is emitted only when the ifunc is called, not just
referenced).
Checked with the elf tests on qemu sparc64 and powerpc64.
Tested-by: Andreas K. Hüttel <dilfridge@gentoo.org>
Add --install option, which copies a pre-built ld.so.cache into place,
honoring the cache and root options and defaults. This gives the user
a canonical "correct" way to install a pre-built cache without risk
of a program trying to load a partially-written file.
Co-authored-by: Adhemerval Zanella Netto <adhemerval.zanella@linaro.org>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
On s390x the test elf/tunconf1 fails with:
tst-tunconf1.c:41: numeric comparison failure (widths 64 and 32)
left: 180388626436 (0x2a00000004); from: (long)perturb
right: 42 (0x2a); from: 42
According to elf/dl-tunables.list, glibc.malloc.perturb is of type int32_t (4byte)
and not size_t (8byte) which was used for TUNABLE_GET_FULL inside the testcase.
Therefore the correct 32bit value 0x2a=42 is written to the to the wrong place
and leads to the comparison failure.
The printf format specifiers for size_t were also adjusted.
Reviewed-by: DJ Delorie <dj@redhat.com>
The per-tunable security level is no longer part of struct _tunable and
no tunable in dl-tunables.list declares one.
Reviewed-by: DJ Delorie <dj@redhat.com>
The environment-variable alias loop in __tunables_init skipped every tunable
whose "initialized" flag was set, which was originally meant only to give
the canonical GLIBC_TUNABLES form precedence over the legacy MALLOC_*
aliases.
Now that the cache also sets "initialized", a legacy alias could no longer
override an *overridable* cache default, even though the canonical
GLIBC_TUNABLES form still could.
Track separately the tunables that were set from GLIBC_TUNABLES during this
call and skip only those in the alias loop
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: DJ Delorie <dj@redhat.com>
_dl_check_ldsocache_needs_loading only stored the stat fields it
compares (mtime, ino, size, dev) on the path where a cache was already
loaded. On the very first call CACHE is NULL and the function returned
"needs loading" without recording those fields, leaving
new_cache_file_time zero. The next call then copied that zero value
into cache_file_time and compared it against the freshly stat'd values,
which always differed, forcing a second, unnecessary load (munmap +
mmap + re-parse) of an unchanged cache at every startup.
It can be shown with repro:
$ cat << EOF > repro.c
#include <dlfcn.h>
int main (void) { dlopen ("does-not-exist-xyz.so.99", RTLD_NOW); return 0; }
EOF
$ gcc repro.c -o repro
$ strace -f -e trace=openat elf/ld.so --library-path . ./repro 2>&1 | grep -c "/etc/ld.so.cache"
The result should be 1, instead of 2.
Record the stat fields as soon as the stat succeeds, before the
CACHE == NULL early return, so the following call has an accurate
baseline and does not spuriously reload.
Reviewed-by: DJ Delorie <dj@redhat.com>
The tunable header signature and version are written by ldconfig but
never checked them on read, so the version field was inert. Reject
the section unless both match.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: DJ Delorie <dj@redhat.com>
_dl_load_cache_tunables bounds each entry's string offsets against
[s_start, start + cache_new->len_strings], but len_strings is an
unvalidated 32-bit field from ld.so.cache and s_start/s_end were int. A
corrupt cache with an oversized len_strings could make s_end exceed the
mapping (or overflow), letting an offset point outside the mmap; the
following strcmp/__strdup would then read unmapped memory.
Compute the offsets as size_t and clamp s_end to cachesize, matching how
the regular library lookup bounds string offsets against the mapping size.
Checked on x86_64-linux-gnu and i686-linux-gnu.
Reviewed-by: DJ Delorie <dj@redhat.com>
The top-level makefile was marked .NOTPARALLEL and ran the
per-subdirectory sub-makes strictly one at a time in the topological
order computed by scripts/gen-sorted.awk. Only the compilations inside
a single subdirectory could run in parallel, so on wide machines every
subdirectory's compile tail and link steps left most cores idle, once
per subdirectory per pass.
Drop .NOTPARALLEL and encode the ordering the serial recursion relied
on as explicit dependencies between the per-subdirectory targets:
* The subdirectories that generate shared files in $(common-objpfx)
consumed by the rest of the build without explicit dependencies run
serially, in their sorted order, before the rest fan out: csu
provides the tree-wide gen-as-const headers, and on Hurd the mach
and hurd directories generate the MiG RPC headers (every other
subdirectory otherwise runs a nested make in hurd/ to create them,
racing under parallel recursion; see sysdeps/mach/hurd/Makefile).
The first of them also materializes the other shared generated files
(abi-versions.h, sysd-syscalls, before-compile headers).
* The edges requested by the Depend files (now emitted by
gen-sorted.awk as subdir-deps-*) are preserved. Edges pointing to
elf are dropped, as the sorted list already overrides them by
forcing elf last.
* The tests and xtests classes only run the per-directory test
programs, which are mutually independent once the others pass has
built the tree. They therefore carry only the others pass barrier
below and none of the csu-first or Depend edges (+ordered_parallel_-
subdir_targets excludes them); otherwise "make subdir/tests" would
also run the tests of every subdirectory reachable through those
edges, rather than just the requested one.
* elf stays last: its rtld link consumes $(common-objpfx)libc_pic.a,
which aggregates every other subdirectory's objects, and its
rtld-Rules recursion compiles into the other subdirectories' object
directories.
* Pass barriers replace the implicit pass ordering: others after lib
(a subdirectory others sub-make would otherwise race to link
libc.so itself), tests/xtests after others, and the testroot
install behind others.
* The threading (nptl, or htl on Hurd) and realtime (rt) tests are
timing-sensitive and were previously shielded from system load by
the global .NOTPARALLEL. With the recursion now parallel, a full
test run ('make check'/'tests', run-built-tests=yes) orders them
after the rest of the test run and one group at a time -- the
threading subdirectory, then rt -- and each serializes its own run
via a .NOTPARALLEL in its Makefile. A targeted 'make subdir/tests'
is not ordered.
The serialization (the per-subdirectory .NOTPARALLEL and the ordering
above) constrains only the test run, not the build of the test
programs; but building and running a subdirectory's tests are fused
in its sub-make, so under run-built-tests=yes the serialized
subdirectories would also build their test programs serially. To
avoid that, the top-level 'make check' (in Makerules) now runs two
passes: it builds every test program with run-built-tests=no, where
the recursion is fully parallel and none of the serialization
applies, and then runs the tests with run-built-tests=yes. 'make
tests' and a subdirectory's own 'check' stay single pass.
* The subdirectory-built files that the top-level libc.so and
linkobj/libc_pic.a rules list as prerequisites (elf/ld.so,
interp.os, sofini.os, sunrpc/librpc_compat_pic.a, and on Hurd
mach/libmachuser_pic.a and hurd/libhurduser_pic.a, from which the
lib*user-link.so inputs of libc.so are built) get order-only edges
on the corresponding sub-make with an explicit empty recipe. A
prerequisite-only rule would trigger an implicit rule search and
this level would compile them itself in the wrong context.
* The install, clean, abi, and stubs target classes keep the
previous total order via a serial dependency chain.
* The elf DSO sorting test recipes, run when make remakes the
included generated makefiles at parse time, create the elf object
directory before writing into it; the serial recursion no longer
guarantees another rule created it first.
* catgets builds locale-specific message catalogs (and tst-catgets
reads one) by running gencat under de_DE.ISO-8859-1, hr_HR.ISO-8859-2
and ja_JP.SJIS, but never declared those locales as prerequisites: it
relied on localedata running before it in the serial order. Under
the parallel recursion gencat could run before localedata generated
the locale, fall back to C, and fail. catgets/Makefile now pulls the
locales in via gen-locales.mk, like the other subdirectories that use
locales in their tests.
Results on a x86_64 machine [1] with default configuration [3]: a
from-scratch build improves from 78.728s to 61s, and check with
run-built-tests=no from 374s to 190s.
On a 80-core aarch64 machine [2] with default configuration [3]: a
from-scratch build improves from 105.251s to 56.703s, and check with
run-built-tests=no from 886.183s to 298.726s.
Build results are unchanged: all 8919 built objects, archives, and
shared objects are bit-identical to the serial build across 7 clean
parallel builds, the installed tree layout is identical, and the
tests.sum failure sets are identical. i686-gnu was verified with
repeated from-scratch builds.
[1] Ryzen 5900x, 12c/24t, gcc 16.1.1, binutils 2.26, and GNU make 4.3
[2] N1, 80c, gcc 15.1.1, binutils 2.25, GNU make 4.3
[3] --enable-stack-protector=all --enable-bind-now=yes --enable-profile=yes
--enable-fortify-source=2 --enable-hardcoded-path-in-tests
Reviewed-by: Sam James <sam@gentoo.org>
Add support for [proc:*] syntax where * matches /proc/self/exe
(fallback: argv[0] unless AT_SECURE). Tunables after such a
line are limited to matching processes.
Note that this filter is reset when including a file or at
end of file.
If the filename starts with a slash (example: [proc:/bin/foo]) the
full path must match. If not (example: [proc:foo]) the basename is
matched.
Add support for filtering out AT_SECURE or non-AT_SECURE binaries:
$glibc.only-for.nonsecure-binaries=1
@glibc.only-for.secure-binaries=1
Reviewed-by: Arjun Shankar <arjun@redhat.com>
Load ld.so.cache and fetch the tunables extension. Apply
those tunables to the current program. We do not yet apply
security policies.
Reviewed-by: Arjun Shankar <arjun@redhat.com>
The purpose of this change is twofold:
1. The ld.so.cache is cached in memory and only re-read if/when
it changes on disk. This allows us to have much more intensive
security checks in the future, without impacting performance as
much. It also allows for cases where the cache is corrupted -
we continue using the last valid one.
2. We break out the load/check logic so that the cache can be
loaded independently of the library lookup, such as for
code that only needs to look at the extensions.
Reviewed-by: Arjun Shankar <arjun@redhat.com>
Adds support for reading /etc/tunables.conf
The file contains one line per tunable, like this:
glibc.foo.bar=14
glibc.malloc.more=0
Additionally, each line can be prefixed with a single word or character
that controls overridability by the GLIBC_TUNABLES env var:
overridable glibc.foo=0
+glibc.foo=0
^ May be overridden (the default)
nonoverridable glibc.foo=0
-glibc.foo=0
^ May not be overridden
The tunable cache format allows for a filter to be assigned to
each tunable, to be used at program start to decide if a tunable
applies to that program. No such filters have yet been specified.
The cache format also stores a pre-parsed value for the tunable, and
the ID of the tunable, to improve load-time performance.
Reviewed-by: Arjun Shankar <arjun@redhat.com>
The current THP segment load approach works only when THP is enabled
with always in the kernel. If THP is enabled with madvise in the
kernel, to enable THP segment load in an application, madvise should
be called with MADV_HUGEPAGE on all THP eligible PT_LOAD segments:
1. Define DL_MAP_DEFAULT_THP_PAGESIZE in hugepages.h and default it to 0.
If DL_MAP_DEFAULT_THP_PAGESIZE is defined, assume kernel THP madvise mode.
If kernel THP mode is always or never, there is an extra madvise call
which has no impact. DL_MAP_DEFAULT_THP_PAGESIZE is defined for x86-64
and 64-bit loongarch.
2. Update _dl_map_segment_align to support madvise THP mode. This fixes
BZ #34079.
3. Call _dl_executable_postprocess in rtld_setup_main_map for dynamic
executables and in LIBC_START_MAIN for static executables, which calls
madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments in
executable. This fixes BZ #34080 for both dynamic and static executables.
4. Call _dl_postprocess_loadcmd_extra in _dl_postprocess_loadcmd, which
calls madvise with MADV_HUGEPAGE on all THP eligible PT_LOAD segments
when loading an object after they have been mapped in. This fixes
BZ #34080 for shared objects.
5. Set the maximum page alignment on THP tests to THP page size as the
default maximum page alignment may be smaller than THP page size.
6. Add tests to verify that large executable PT_LOAD segments in
executables are mapped at addresses aligned to THP page size when the
kernel is configured to use THP in "always" mode or "madvise" mode by
inspecting /proc/self/maps to check that the mapping address is aligned
to THP page size reported by the kernel. Also verify that madvise is
called with MADV_HUGEPAGE when the glibc tunable glibc.elf.thp=1 is used
and madvise isn't called with MADV_HUGEPAGE when the glibc tunable
glibc.elf.thp=0 is used.
Skip these tests if THP page size cannot be determined or if THP is not
enabled in "always" mode nor "madvise" mode.
Quote WANG Rui <wangrui@loongson.cn>:
From benchmarking a clang build of the Linux kernel on x86_64 with
your patch in THP madvise mode, I observed that iTLB misses were
reduced, similar to what we see in THP always mode.
NB: Some THP tests fail on arm due to limitations of arm32 kABI:
https://sourceware.org/bugzilla/show_bug.cgi?id=34096
Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Commit a5ec880f80 ("Consolidate pointer
guard to use a relro variable instead of the TCB") moved the pointer
guard to the module-local relro variable __pointer_chk_guard_local and
dropped the THREAD_{SET,COPY}_POINTER_GUARD macros, which are no longer
defined by any target.
Checked on a build for all afftected targets.
Once the pointer and stack guards have been derived from AT_RANDOM, scrub
the bytes and refill them with new random data unrelated to the guards.
On Linux, it uses getrandom syscall (as for tcache_key_initialize), and
fallback to zero the memory if the syscall is not avaiable.
This keeps AT_RANDOM useful to applications while ensuring those bytes no
longer reveal the guards.
The work is done by _dl_reseed_random, called once the guards are in place
and before any ELF constructor can observe AT_RANDOM: in security_init for
the dynamic loader and in __libc_start_main for statically linked programs.
Checked on aarch64-linux-gnu, x86_64-linux-gnu, and i686-linux-gnu.
Reviewed-by: DJ Delorie <dj@redhat.com>
The static-dlopen does not initialize the pointer guard for ABIs that
define THREAD_SET_POINTER_GUARD. Besides not properly guard the
pointer if a libc.so symbol is called, this can lead to setjmp
failures (a jmp_buf set up by the loaded libc.so.6 cannot be restored
by the static program's __longjmp, and vice versa).
Seed the just-mapped loader's __pointer_chk_guard from the program's
__pointer_chk_guard_local in __rtld_static_init, next to the other
runtime values copied there.
Checked on aarch64-linux-gnu, x86_64-linux-gnu, and i686-linux-gnu.
Reviewed-by: DJ Delorie <dj@redhat.com>
An IFUNC resolver in a DSO that is being loaded by dlopen is allowed
to read its own TLS storage during the resolver call. After
af34b1376a ("elf: Initialize static TLS before relocation processing
(BZ 34164)") that works for the initial-exec model on every supported
architecture.
However, it does not work for the dynamic-TLS path (on both -mtls-dialect mode,
if the ABI supports both). Both lookup paths index the calling thread's
DTV by the new module's l_tls_modid and, on miss, walk
GL(dl_tls_dtv_slotinfo_list) to discover the module and lazily allocate
its TLS block. The just-loaded DSO is however not yet in that list when
its resolver fires, so the lookup faults inside dlopen. This is the
direct dlopen analog of BZ 34164.
The solution is to reorder dl_open_worker_begin so the slotinfo install
happens before the relocation pass. The new order is:
1. resize_scopes, resize_tls_slotinfo, add_to_global_resize
(unchanged, still recoverable).
2. update_tls_slotinfo: register the new modules in slotinfo, bump
dl_tls_generation, initialise their static TLS images.
3. Relocate the new objects. IFUNC resolvers can now read their
own DSO's __thread storage via any TLS model.
4. Demarcation point.
5. update_scopes, _dl_find_object_update.
Checked on aarch64-linux-gnu and x86_64-linux-gnu.
Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
The test failed with --disable-default-pie because its primary check read
the resolver's diagnostic side effect before the resolver had run.
In a non-PIE executable the references to the IFUNC (fptr and ifunc_tls)
are satisfied through a canonical IPLT entry in the executable itself.
Under the default lazy binding that IPLT is resolved on first use, not
during startup relocation, so the resolver had not yet run when
'check_sentinel' inspected that value. With a PIE executable
(or LD_BIND_NOW=1) the resolver runs eagerly at startup and the check
passed. The dlopen path was unaffected because dlopen resolves the
data relocation eagerly.
This is a test ordering issue: the resolver always reads the initial-exec
TLS correctly whenever it runs, so the BZ 34164 fix is not involved.
Reorder test_tls_ifunc so that fptr/ifunc_tls force the IFUNC to be
resolved before the last_seen_sentinel value is inspected.
Checked on x86_64-linux-gnu and i686-linux-gnu with --disable-default-pie.
Reported-by: Yury Khrustalev <yury.khrustalev@arm.com>
Reviewed-by: Yury Khrustalev <yury.khrustalev@arm.com>
Remove the --enable-memory-tagging configure option along with
all associated variables and macros.
Removing the glibc.mem.tagging tunable.
Remove the memory-tagging makefile variable.
Remove the USE_MTAG macro definition and code that is conditionally
compiled when this macro is defined.
As a result, we change 'mtag_mmap_flags' to 'extra_mmap_prot' that
is now always defined. Change of the name due to this being used
as part of PROT options in mmap syscalls rather than part of flags.
Remove 'mtag_enabled' that would become compile-time false. Also
remove any code that would never be compiled when 'mtag_enabled'
is false.
Remove AArch64-specific code pertinent to memory tagging, that is
currently broken, from the core malloc implementation. We keep
the assembly code, since we are going to need it in the future. to
preserve Git history.
Reviewed-by: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
When the compiler defines __ARM_BUILDATTR64_FV, emit AArch64
feature-and-bits build attributes for BTI, PAC, and GCS from sysdep.h
instead of a GNU property note. Keep the GNU property note as the
fallback for older toolchains.
Mirror the same marking logic in elf/tst-asm-helper.h so custom test
DSOs and assembly tests that cannot include sysdep.h get consistent
feature marking.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
The af34b1376a (BZ 34164) changed the
TLS setup from:
relocation loop (applies relocations to .tdata in DSO memory)
_dl_allocate_tls_init copies relocated .tdata -> main thread TLS
to a new order:
_dl_allocate_tls_init copies unrelocated .tdata -> main thread TLS
relocation loop (relocates .tdata in DSO memory, but the TLS block
has stale copies)
This broke file-scope thread-local initialised with the address of a
function (for instance the cache structs in libmpfr).
Fix it by splitting ELF_DYNAMIC_RELOCATE inside
_dl_relocate_object_no_relro into the non-IRELATIVE and IRELATIVE
sub-passes (similar as done on static-pie startup by b75ad99d45)
and call _dl_init_static_tls between them. By the time the IFUNC
pass fires, .tdata is fully relocated.
Checked on x86_64-linux-gnu and aarch64-linux-gnu.
Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
In static linking the IFUNC IPLT (apply_irel for non-PIE, the IRELATIVE
phase inside _dl_relocate_static_pie for static-pie) ran before
__libc_setup_tls and before _dl_setup_stack_chk_guard. When a resolver
is compiled with -fstack-protector(-all) its prologue loads the canary
from the TCB (TCB-canary ABIs: x86_64, i386, powerpc, s390) or from
__stack_chk_guard (global-var ABIs). On the former the resolver
crashed reading an unmapped TCB; on the latter it loaded a zero canary
(no crash, but the check is ineffective). The same applies to a
resolver that reads any thread-local: it crashes on TCB-canary ABIs and
observes a zero-filled slot on the others (BZ 20680). The pointer
guard has the same problem (e.g. resolvers that register an atexit
handler).
Reorder csu/libc-start.c so that ARCH_SETUP_TLS, the stack-protector
canary and the pointer guard are set up before any IFUNC resolver
runs. For static-pie this requires splitting the existing
_dl_relocate_static_pie into two phases so the TCB/canary setup can be
interleaved between the non-IRELATIVE and IRELATIVE passes.
The historical ARCH_SETUP_IREL / ARCH_APPLY_IREL split (introduced for
powerpc so its IFUNC resolvers could read TCB fields like hwcap and
at_platform) is no longer required: TLS is now set up before either
macro runs. ARCH_APPLY_IREL is removed, ARCH_SETUP_IREL does the work
uniformly on every arch, and the powerpc-specific libc-start.h becomes
redundant.
__libc_setup_tls reaches memcpy / mempcpy via _dl_allocate_tls_init in
elf/dl-tls.c, so it requires update ABI specific dl-symbol-redir-ifunc.h
with memcpy/memmove.
Tests added (each fails pre-fix on TCB-canary ABIs with SIGSEGV; the
static-protector variants additionally fail on global-var ABIs with a
"resolver_canary != main_canary" diagnostic):
elf/tst-ifunc-bz28817 static-pie + TLS in
resolver (BZ 28817)
elf/tst-ifunc-resolver-protector dynamic
elf/tst-ifunc-resolver-protector-static static-pie
elf/tst-ifunc-resolver-protector-static-non-pie non-PIE static
Checked on aarch64-linux-gnu, arm-linux-gnueabihf, x86_64-linux-gnu,
and i686-linux-gnu
I also ran the ELF tests on qemu system for loongarch64-linux-gnuf64,
powerpc-linux-gnu, powerpc-linux-gnu-power4, powerpc-linux-gnu-soft,
powerpc64-linux-gnu, powerpc64le-linux-gnu, riscv64-linux-gnu, and
s390x-linux-gnu.
Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
An IFUNC resolver firing during dynamic linker relocation reads its
DSO's __thread storage from a zero-filled slot: init_tls() allocates
the static TLS block zero-filled, but .tdata is not copied in until
the trailing _dl_allocate_tls_init at the end of dl_main, long after
the per-object phase 2 resolvers from commit 63b31c05a8 have run.
A resolver that *writes* TLS is even worse off -- the write is
clobbered by that same trailing copy.
dl_main (elf/rtld.c): populate the DTV slotinfo, bump
dl_tls_generation, and call _dl_allocate_tls_init right after
init_tls(), before the relocation loop.
_dl_try_allocate_static_tls (elf/dl-reloc.c): drop the
"defer-if-not-relocated" branch and always run _dl_init_static_tls
inline, so a CHECK_STATIC_TLS allocation triggered mid-relocation
initialises the slot before the same object's phase 2 fires.
The new tests cheks some scenarios:
elf/tst-ifunc-tls-init resolver reads its DSO's IE TLS.
elf/tst-ifunc-tls-init-dlopen same, via dlopen.
elf/tst-ifunc-tls-write resolver write to TLS must survive
to main.
Checked on aarch64-linux-gnu, x86_64-linux-gnu, and i686-linux-gnu.
Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
Adhemerval Zanella
H.J. Lu
litenglong
Matt Turner
Sam James
Frédéric Bérat
Xi Ruoyao
DJ DelorieandAdhemerval Zanella Netto
Stefan Liebler
Yury Khrustalev
Muhammad Kamran