glibc/sysdeps/unix/sysv/linux/x86/lowlevellock.h

112 lines
3.7 KiB
C
Raw Normal View History

x86: Remove arch-specific low level lock implementation This patch removes the arch-specific x86 assembly implementation for low level locking and consolidate both 64 bits and 32 bits in a single implementation. Different than other architectures, x86 lll_trylock, lll_lock, and lll_unlock implements a single-thread optimization to avoid atomic operation, using cmpxchgl instead. This patch implements by using the new single-thread.h definitions in a generic way, although using the previous semantic. The lll_cond_trylock, lll_cond_lock, and lll_timedlock just use atomic operations plus calls to lll_lock_wait*. For __lll_lock_wait_private and __lll_lock_wait the generic implemtation there is no indication that assembly implementation is required performance-wise. Checked on x86_64-linux-gnu and i686-linux-gnu. * sysdeps/nptl/lowlevellock.h (__lll_trylock): New macro. (lll_trylock): Call __lll_trylock. * sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S: Remove file. * sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: New file. * sysdeps/unix/sysv/linux/x86_64/cancellation.S: Include lowlevellock-futex.h.
2019-02-21 16:33:54 +00:00
/* Low-level lock implementation, x86 version.
Copyright (C) 2019-2020 Free Software Foundation, Inc.
x86: Remove arch-specific low level lock implementation This patch removes the arch-specific x86 assembly implementation for low level locking and consolidate both 64 bits and 32 bits in a single implementation. Different than other architectures, x86 lll_trylock, lll_lock, and lll_unlock implements a single-thread optimization to avoid atomic operation, using cmpxchgl instead. This patch implements by using the new single-thread.h definitions in a generic way, although using the previous semantic. The lll_cond_trylock, lll_cond_lock, and lll_timedlock just use atomic operations plus calls to lll_lock_wait*. For __lll_lock_wait_private and __lll_lock_wait the generic implemtation there is no indication that assembly implementation is required performance-wise. Checked on x86_64-linux-gnu and i686-linux-gnu. * sysdeps/nptl/lowlevellock.h (__lll_trylock): New macro. (lll_trylock): Call __lll_trylock. * sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S: Remove file. * sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: New file. * sysdeps/unix/sysv/linux/x86_64/cancellation.S: Include lowlevellock-futex.h.
2019-02-21 16:33:54 +00:00
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
Prefer https to http for gnu.org and fsf.org URLs Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
<https://www.gnu.org/licenses/>. */
x86: Remove arch-specific low level lock implementation This patch removes the arch-specific x86 assembly implementation for low level locking and consolidate both 64 bits and 32 bits in a single implementation. Different than other architectures, x86 lll_trylock, lll_lock, and lll_unlock implements a single-thread optimization to avoid atomic operation, using cmpxchgl instead. This patch implements by using the new single-thread.h definitions in a generic way, although using the previous semantic. The lll_cond_trylock, lll_cond_lock, and lll_timedlock just use atomic operations plus calls to lll_lock_wait*. For __lll_lock_wait_private and __lll_lock_wait the generic implemtation there is no indication that assembly implementation is required performance-wise. Checked on x86_64-linux-gnu and i686-linux-gnu. * sysdeps/nptl/lowlevellock.h (__lll_trylock): New macro. (lll_trylock): Call __lll_trylock. * sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S: Remove file. * sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: New file. * sysdeps/unix/sysv/linux/x86_64/cancellation.S: Include lowlevellock-futex.h.
2019-02-21 16:33:54 +00:00
#ifndef _X86_64_LOWLEVELLOCK_H
#define _X86_64_LOWLEVELLOCK_H 1
#ifndef __ASSEMBLER__
#include <sysdeps/nptl/lowlevellock.h>
#include <single-thread.h>
/* The lll_trylock, lll_lock, and lll_unlock implements a single-thread
optimization using the cmpxchgl instruction. It checks if the process
is single thread and avoid a more expensive atomic instruction. */
/* The single-thread optimization only works for libc itself, we need
atomicity for libpthread in case of shared futexes. */
#if !IS_IN(libc)
# define is_single_thread 0
#else
# define is_single_thread SINGLE_THREAD_P
#endif
/* In the __lllc_as we simply return the value in %eax after the cmpxchg
instruction. In case the operation succeeded this value is zero. In case
the operation failed, the cmpxchg instruction has loaded the current value
of the memory work which is guaranteed to be nonzero. */
static inline int
__attribute__ ((always_inline))
__lll_cas_lock (int *futex)
{
int ret;
asm volatile ("cmpxchgl %2, %1"
: "=a" (ret), "=m" (*futex)
: "r" (1), "m" (*futex), "0" (0)
: "memory");
return ret;
}
#undef lll_trylock
#define lll_trylock(lock) \
({ \
int __ret; \
if (is_single_thread) \
__ret = __lll_cas_lock (&(lock)); \
else \
__ret = __lll_trylock (&(lock)); \
__ret; \
})
#undef lll_lock
#define lll_lock(lock, private) \
((void) \
({ \
if (is_single_thread) \
__lll_cas_lock (&(lock)); \
else \
__lll_lock (&(lock), private); \
}))
#undef lll_unlock
#define lll_unlock(lock, private) \
((void) \
({ \
if (is_single_thread) \
(lock)--; \
else \
__lll_unlock (&(lock), private); \
}))
nptl: Rename lll_timedlock to lll_clocklock and add clockid parameter Rename lll_timedlock to lll_clocklock and add clockid parameter to indicate the clock that the abstime parameter should be measured against in preparation for adding pthread_mutex_clocklock. The name change mirrors the naming for the exposed pthread functions: timed => absolute timeout measured against CLOCK_REALTIME (or clock specified by attribute in the case of pthread_cond_timedwait.) clock => absolute timeout measured against clock specified in preceding parameter. * sysdeps/nptl/lowlevellock.h (lll_clocklock): Rename from lll_timedlock and add clockid parameter. (__lll_clocklock): Rename from __lll_timedlock and add clockid parameter. * sysdeps/unix/sysv/linux/sparc/lowlevellock.h (lll_clocklock): Likewise. * nptl/lll_timedlock_wait.c (__lll_clocklock_wait): Rename from __lll_timedlock_wait and add clockid parameter. Use __clock_gettime rather than __gettimeofday so that clockid can be used. This means that conversion from struct timeval is no longer required. * sysdeps/sparc/sparc32/lowlevellock.c (lll_clocklock_wait): Likewise. * sysdeps/sparc/sparc32/lll_timedlock_wait.c: Update comment to refer to __lll_clocklock_wait rather than __lll_timedlock_wait. * nptl/pthread_mutex_timedlock.c (lll_clocklock_elision): Rename from lll_timedlock_elision, add clockid parameter and use meaningful names for other parameters. (__pthread_mutex_timedlock): Pass CLOCK_REALTIME where necessary to lll_clocklock and lll_clocklock_elision. * sysdeps/unix/sysv/linux/powerpc/lowlevellock.h (lll_clocklock_elision): Rename from lll_timedlock_elision and add clockid parameter. (__lll_clocklock_elision): Rename from __lll_timedlock_elision and add clockid parameter. * sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/powerpc/elision-timed.c (__lll_lock_elision): Call __lll_clocklock_elision rather than __lll_timedlock_elision. (EXTRAARG): Add clockid parameter. (LLL_LOCK): Likewise. * sysdeps/unix/sysv/linux/s390/elision-timed.c: Likewise. * sysdeps/unix/sysv/linux/x86/elision-timed.c: Likewise. Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
2019-06-24 13:08:25 +00:00
extern int __lll_clocklock_elision (int *futex, short *adapt_count,
clockid_t clockid,
const struct timespec *timeout,
int private) attribute_hidden;
x86: Remove arch-specific low level lock implementation This patch removes the arch-specific x86 assembly implementation for low level locking and consolidate both 64 bits and 32 bits in a single implementation. Different than other architectures, x86 lll_trylock, lll_lock, and lll_unlock implements a single-thread optimization to avoid atomic operation, using cmpxchgl instead. This patch implements by using the new single-thread.h definitions in a generic way, although using the previous semantic. The lll_cond_trylock, lll_cond_lock, and lll_timedlock just use atomic operations plus calls to lll_lock_wait*. For __lll_lock_wait_private and __lll_lock_wait the generic implemtation there is no indication that assembly implementation is required performance-wise. Checked on x86_64-linux-gnu and i686-linux-gnu. * sysdeps/nptl/lowlevellock.h (__lll_trylock): New macro. (lll_trylock): Call __lll_trylock. * sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S: Remove file. * sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: New file. * sysdeps/unix/sysv/linux/x86_64/cancellation.S: Include lowlevellock-futex.h.
2019-02-21 16:33:54 +00:00
nptl: Rename lll_timedlock to lll_clocklock and add clockid parameter Rename lll_timedlock to lll_clocklock and add clockid parameter to indicate the clock that the abstime parameter should be measured against in preparation for adding pthread_mutex_clocklock. The name change mirrors the naming for the exposed pthread functions: timed => absolute timeout measured against CLOCK_REALTIME (or clock specified by attribute in the case of pthread_cond_timedwait.) clock => absolute timeout measured against clock specified in preceding parameter. * sysdeps/nptl/lowlevellock.h (lll_clocklock): Rename from lll_timedlock and add clockid parameter. (__lll_clocklock): Rename from __lll_timedlock and add clockid parameter. * sysdeps/unix/sysv/linux/sparc/lowlevellock.h (lll_clocklock): Likewise. * nptl/lll_timedlock_wait.c (__lll_clocklock_wait): Rename from __lll_timedlock_wait and add clockid parameter. Use __clock_gettime rather than __gettimeofday so that clockid can be used. This means that conversion from struct timeval is no longer required. * sysdeps/sparc/sparc32/lowlevellock.c (lll_clocklock_wait): Likewise. * sysdeps/sparc/sparc32/lll_timedlock_wait.c: Update comment to refer to __lll_clocklock_wait rather than __lll_timedlock_wait. * nptl/pthread_mutex_timedlock.c (lll_clocklock_elision): Rename from lll_timedlock_elision, add clockid parameter and use meaningful names for other parameters. (__pthread_mutex_timedlock): Pass CLOCK_REALTIME where necessary to lll_clocklock and lll_clocklock_elision. * sysdeps/unix/sysv/linux/powerpc/lowlevellock.h (lll_clocklock_elision): Rename from lll_timedlock_elision and add clockid parameter. (__lll_clocklock_elision): Rename from __lll_timedlock_elision and add clockid parameter. * sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/powerpc/elision-timed.c (__lll_lock_elision): Call __lll_clocklock_elision rather than __lll_timedlock_elision. (EXTRAARG): Add clockid parameter. (LLL_LOCK): Likewise. * sysdeps/unix/sysv/linux/s390/elision-timed.c: Likewise. * sysdeps/unix/sysv/linux/x86/elision-timed.c: Likewise. Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
2019-06-24 13:08:25 +00:00
#define lll_clocklock_elision(futex, adapt_count, clockid, timeout, private) \
__lll_clocklock_elision (&(futex), &(adapt_count), clockid, timeout, private)
x86: Remove arch-specific low level lock implementation This patch removes the arch-specific x86 assembly implementation for low level locking and consolidate both 64 bits and 32 bits in a single implementation. Different than other architectures, x86 lll_trylock, lll_lock, and lll_unlock implements a single-thread optimization to avoid atomic operation, using cmpxchgl instead. This patch implements by using the new single-thread.h definitions in a generic way, although using the previous semantic. The lll_cond_trylock, lll_cond_lock, and lll_timedlock just use atomic operations plus calls to lll_lock_wait*. For __lll_lock_wait_private and __lll_lock_wait the generic implemtation there is no indication that assembly implementation is required performance-wise. Checked on x86_64-linux-gnu and i686-linux-gnu. * sysdeps/nptl/lowlevellock.h (__lll_trylock): New macro. (lll_trylock): Call __lll_trylock. * sysdeps/unix/sysv/linux/i386/libc-lowlevellock.S: Remove file. * sysdeps/unix/sysv/linux/i386/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lll_timedlock_wait.c: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/x86/lowlevellock.h: New file. * sysdeps/unix/sysv/linux/x86_64/cancellation.S: Include lowlevellock-futex.h.
2019-02-21 16:33:54 +00:00
extern int __lll_lock_elision (int *futex, short *adapt_count, int private)
attribute_hidden;
extern int __lll_unlock_elision (int *lock, int private)
attribute_hidden;
extern int __lll_trylock_elision (int *lock, short *adapt_count)
attribute_hidden;
#define lll_lock_elision(futex, adapt_count, private) \
__lll_lock_elision (&(futex), &(adapt_count), private)
#define lll_unlock_elision(futex, adapt_count, private) \
__lll_unlock_elision (&(futex), private)
#define lll_trylock_elision(futex, adapt_count) \
__lll_trylock_elision (&(futex), &(adapt_count))
#endif /* !__ASSEMBLER__ */
#endif /* lowlevellock.h */