qiurui
/
glibc
mirror of git://sourceware.org/git/glibc.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
glibc/sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c

445 lines
16 KiB
C
Raw Normal View History

PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
/* Enumerate available IFUNC implementations of a function. PowerPC64 version.
Update copyright dates with scripts/update-copyrights
2023-01-06 21:08:04 +00:00
Copyright (C) 2013-2023 Free Software Foundation, Inc.
PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
Prefer https to http for gnu.org and fsf.org URLs Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
2019-09-07 05:40:42 +00:00
<https://www.gnu.org/licenses/>. */
PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
#include <assert.h>
PowerPC: Influence cpu/arch hwcap features via GLIBC_TUNABLES This patch enables the option to influence hwcaps used by PowerPC. The environment variable, GLIBC_TUNABLES=glibc.cpu.hwcaps=-xxx,yyy,-zzz...., can be used to enable CPU/ARCH feature yyy, disable CPU/ARCH feature xxx and zzz, where the feature name is case-sensitive and has to match the ones mentioned in the file{sysdeps/powerpc/dl-procinfo.c}. Note that the hwcap tunables only used in the IFUNC selection. Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
2023-08-01 12:41:17 +00:00
#include <cpu-features.h>
PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
#include <string.h>
#include <wchar.h>
#include <ldsodefs.h>
#include <ifunc-impl-list.h>
size_t
__libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array,
size_t max)
{
Add bounds check to __libc_ifunc_impl_list Add a proper bounds check to __libc_ifunc_impl_list. This makes MAX_IFUNC redundant and fixes several targets that will write outside the array. To avoid unnecessary large diffs, pass the maximum in the argument 'i' to IFUNC_IMPL_ADD - 'max' can be used in new ifunc definitions and existing ones can be updated if desired. Passes buildmanyglibc. Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
2022-06-10 16:13:29 +00:00
size_t i = max;
PowerPC: Influence cpu/arch hwcap features via GLIBC_TUNABLES This patch enables the option to influence hwcaps used by PowerPC. The environment variable, GLIBC_TUNABLES=glibc.cpu.hwcaps=-xxx,yyy,-zzz...., can be used to enable CPU/ARCH feature yyy, disable CPU/ARCH feature xxx and zzz, where the feature name is case-sensitive and has to match the ones mentioned in the file{sysdeps/powerpc/dl-procinfo.c}. Note that the hwcap tunables only used in the IFUNC selection. Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
2023-08-01 12:41:17 +00:00
const struct cpu_features *features = &GLRO(dl_powerpc_cpu_features);
unsigned long int hwcap = features->hwcap;
unsigned long int hwcap2 = features->hwcap2;
powerpc64: Check cacheline size before using optimised memset routines A number of optimised memset routines assume the cacheline size is 128B, so we better check before using them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:50 +00:00
#ifdef SHARED
int cacheline_size = GLRO(dl_cache_line_size);
#endif
PowerPC: memset optimization for POWER8/PPC64 This patch adds an optimized memset implementation for POWER8. For sizes from 0 to 255 bytes, a word/doubleword algorithm similar to POWER7 optimized one is used. For size higher than 255 two strategies are used: 1. If the constant is different than 0, the memory is written with altivec vector instruction; 2. If constant is 0, dbcz instructions are used. The loop is unrolled to clear 512 byte at time. Using vector instructions increases throughput considerable, with a double performance for sizes larger than 1024. The dcbz loops unrolls also shows performance improvement, by doubling throughput for sizes larger than 8192 bytes.
2014-07-15 16:19:09 +00:00
PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
/* hwcap contains only the latest supported ISA, the code checks which is
and fills the previous supported ones. */
if (hwcap & PPC_FEATURE_ARCH_2_06)
hwcap |= PPC_FEATURE_ARCH_2_05 | PPC_FEATURE_POWER5_PLUS |
PPC_FEATURE_POWER5 | PPC_FEATURE_POWER4;
else if (hwcap & PPC_FEATURE_ARCH_2_05)
hwcap |= PPC_FEATURE_POWER5_PLUS | PPC_FEATURE_POWER5 | PPC_FEATURE_POWER4;
else if (hwcap & PPC_FEATURE_POWER5_PLUS)
hwcap |= PPC_FEATURE_POWER5 | PPC_FEATURE_POWER4;
else if (hwcap & PPC_FEATURE_POWER5)
hwcap |= PPC_FEATURE_POWER4;
#ifdef SHARED
/* Support sysdeps/powerpc/powerpc64/multiarch/memcpy.c. */
IFUNC_IMPL (i, name, memcpy,
powerpc64le: Optimize memcpy for POWER10 This implementation is based on __memcpy_power8_cached and integrates suggestions from Anton Blanchard. It benefits from loads and stores with length for short lengths and for tail code, simplifying the code. All unaligned memory accesses use instructions that do not generate alignment interrupts on POWER10, making it safe to use on caching-inhibited memory. The main loop has also been modified in order to increase instruction throughput by reducing the dependency on updates from previous iterations. On average, this implementation provides around 30% improvement when compared to __memcpy_power7 and 10% improvement in comparison to __memcpy_power8_cached.
2021-04-30 21:12:08 +00:00
#ifdef __LITTLE_ENDIAN__
IFUNC_IMPL_ADD (array, i, memcpy,
hwcap2 & PPC_FEATURE2_ARCH_3_1
&& hwcap & PPC_FEATURE_HAS_VSX,
__memcpy_power10)
#endif
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, memcpy, hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: POWER8 memcpy optimization for cached memory On POWER8, unaligned memory accesses to cached memory has little impact on performance as opposed to its ancestors. It is disabled by default and will only be available when the tunable glibc.tune.cached_memopt is set to 1. __memcpy_power8_cached __memcpy_power7 ============================================================ max-size=4096: 33325.70 ( 12.65%) 38153.00 max-size=8192: 32878.20 ( 11.17%) 37012.30 max-size=16384: 33782.20 ( 11.61%) 38219.20 max-size=32768: 33296.20 ( 11.30%) 37538.30 max-size=65536: 33765.60 ( 10.53%) 37738.40 * manual/tunables.texi (Hardware Capability Tunables): Document glibc.tune.cached_memopt. * sysdeps/powerpc/cpu-features.c: New file. * sysdeps/powerpc/cpu-features.h: New file. * sysdeps/powerpc/dl-procinfo.c [!IS_IN(ldconfig)]: Add _dl_powerpc_cpu_features. * sysdeps/powerpc/dl-tunables.list: New file. * sysdeps/powerpc/ldsodefs.h: Include cpu-features.h. * sysdeps/powerpc/powerpc32/power4/multiarch/init-arch.h (INIT_ARCH): Initialize use_aligned_memopt. * sysdeps/powerpc/powerpc64/dl-machine.h [defined(SHARED && IS_IN(rtld))]: Restrict dl_platform_init availability and initialize CPU features used by tunables. * sysdeps/powerpc/powerpc64/multiarch/Makefile (sysdep_routines): Add memcpy-power8-cached. * sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c: Add __memcpy_power8_cached. * sysdeps/powerpc/powerpc64/multiarch/memcpy.c: Likewise. * sysdeps/powerpc/powerpc64/multiarch/memcpy-power8-cached.S: New file. Reviewed-by: Rajalakshmi Srinivasaraghavan <raji@linux.vnet.ibm.com>
2017-12-11 19:39:42 +00:00
__memcpy_power8_cached)
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, memcpy, hwcap & PPC_FEATURE_ARCH_2_06
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
__memcpy_power7)
IFUNC_IMPL_ADD (array, i, memcpy, hwcap & PPC_FEATURE_ARCH_2_06,
__memcpy_a2)
IFUNC_IMPL_ADD (array, i, memcpy, hwcap & PPC_FEATURE_ARCH_2_05,
__memcpy_power6)
IFUNC_IMPL_ADD (array, i, memcpy, hwcap & PPC_FEATURE_CELL_BE,
__memcpy_cell)
IFUNC_IMPL_ADD (array, i, memcpy, hwcap & PPC_FEATURE_POWER4,
__memcpy_power4)
IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_ppc))
PowerPC: multiarch memset/bzero for PowerPC64
2013-12-13 19:33:16 +00:00
PowerPC: optimized memmove for POWER7/PPC64 This patch adds an optimized memmove optimization for POWER7/powerpc64. Basically the idea is to use the memcpy for POWER7 on non-overlapped memory regions and a optimized backward memcpy for memory regions that overlap (similar to the idea of string/memmove.c). The backward memcpy algorithm used is similar the one use for memcpy for POWER7, with adjustments done for alignment. The difference is memory is always aligned to 16 bytes before using VSX/altivec instructions.
2014-06-20 17:55:16 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/memmove.c. */
IFUNC_IMPL (i, name, memmove,
powerpc64le: Optimized memmove for POWER10 This patch was initially based on the __memmove_power7 with some ideas from strncpy implementation for Power 9. Improvements from __memmove_power7: 1. Use lxvl/stxvl for alignment code. The code for Power 7 uses branches when the input is not naturally aligned to the width of a vector. The new implementation uses lxvl/stxvl instead which reduces pressure on GPRs. It also allows the removal of branch instructions, implicitly removing branch stalls and mispredictions. 2. Use of lxv/stxv and lxvl/stxvl pair is safe to use on Cache Inhibited memory. On Power 10 vector load and stores are safe to use on CI memory for addresses unaligned to 16B. This code takes advantage of this to do unaligned loads. The unaligned loads don't have a significant performance impact by themselves. However doing so decreases register pressure on GPRs and interdependence stalls on load/store pairs. This also improved readability as there are now less code paths for different alignments. Finally this reduces the overall code size. 3. Improved performance. This version runs on average about 30% better than memmove_power7 for lengths larger than 8KB. For input lengths shorter than 8KB the improvement is smaller, it has on average about 17% better performance. This version has a degradation of about 50% for input lengths in the 0 to 31 bytes range when dest is unaligned. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-04-30 21:12:08 +00:00
#ifdef __LITTLE_ENDIAN__
IFUNC_IMPL_ADD (array, i, memmove,
powerpc64le: Fix ifunc selection for memset, memmove, bzero and bcopy The hwcap2 check for the aforementioned functions should check for both PPC_FEATURE2_ARCH_3_1 and PPC_FEATURE2_HAS_ISEL but was mistakenly checking for any one of them, enabling isa 3.1 version of the functions in incompatible processors, like POWER8. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-05-03 19:59:35 +00:00
hwcap2 & PPC_FEATURE2_ARCH_3_1
&& hwcap2 & PPC_FEATURE2_HAS_ISEL
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc64le: Optimized memmove for POWER10 This patch was initially based on the __memmove_power7 with some ideas from strncpy implementation for Power 9. Improvements from __memmove_power7: 1. Use lxvl/stxvl for alignment code. The code for Power 7 uses branches when the input is not naturally aligned to the width of a vector. The new implementation uses lxvl/stxvl instead which reduces pressure on GPRs. It also allows the removal of branch instructions, implicitly removing branch stalls and mispredictions. 2. Use of lxv/stxv and lxvl/stxvl pair is safe to use on Cache Inhibited memory. On Power 10 vector load and stores are safe to use on CI memory for addresses unaligned to 16B. This code takes advantage of this to do unaligned loads. The unaligned loads don't have a significant performance impact by themselves. However doing so decreases register pressure on GPRs and interdependence stalls on load/store pairs. This also improved readability as there are now less code paths for different alignments. Finally this reduces the overall code size. 3. Improved performance. This version runs on average about 30% better than memmove_power7 for lengths larger than 8KB. For input lengths shorter than 8KB the improvement is smaller, it has on average about 17% better performance. This version has a degradation of about 50% for input lengths in the 0 to 31 bytes range when dest is unaligned. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-04-30 21:12:08 +00:00
__memmove_power10)
#endif
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, memmove, hwcap & PPC_FEATURE_ARCH_2_06
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
PowerPC: optimized memmove for POWER7/PPC64 This patch adds an optimized memmove optimization for POWER7/powerpc64. Basically the idea is to use the memcpy for POWER7 on non-overlapped memory regions and a optimized backward memcpy for memory regions that overlap (similar to the idea of string/memmove.c). The backward memcpy algorithm used is similar the one use for memcpy for POWER7, with adjustments done for alignment. The difference is memory is always aligned to 16 bytes before using VSX/altivec instructions.
2014-06-20 17:55:16 +00:00
__memmove_power7)
IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_ppc))
PowerPC: multiarch memset/bzero for PowerPC64
2013-12-13 19:33:16 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/memset.c. */
IFUNC_IMPL (i, name, memset,
powerpc64le: Optimize memset for POWER10 This implementation is based on __memset_power8 and integrates a lot of suggestions from Anton Blanchard. The biggest difference is that it makes extensive use of stxvl to alignment and tail code to avoid branches and small stores. It has three main execution paths: a) "Short lengths" for lengths up to 64 bytes, avoiding as many branches as possible. b) "General case" for larger lengths, it has an alignment section using stxvl to avoid branches, a 128 bytes loop and then a tail code, again using stxvl with few branches. c) "Zeroing cache blocks" for lengths from 256 bytes upwards and set value being zero. It is mostly the __memset_power8 code but the alignment phase was simplified because, at this point, address is already 16-bytes aligned and also changed to use vector stores. The tail code was also simplified to reuse the general case tail. All unaligned stores use stxvl instructions that do not generate alignment interrupts on POWER10, making it safe to use on caching-inhibited memory. On average, this implementation provides something around 30% improvement when compared to __memset_power8. Reviewed-by: Matheus Castanho <msc@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-04-30 21:12:08 +00:00
#ifdef __LITTLE_ENDIAN__
IFUNC_IMPL_ADD (array, i, memset,
powerpc64le: Fix ifunc selection for memset, memmove, bzero and bcopy The hwcap2 check for the aforementioned functions should check for both PPC_FEATURE2_ARCH_3_1 and PPC_FEATURE2_HAS_ISEL but was mistakenly checking for any one of them, enabling isa 3.1 version of the functions in incompatible processors, like POWER8. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-05-03 19:59:35 +00:00
hwcap2 & PPC_FEATURE2_ARCH_3_1
&& hwcap2 & PPC_FEATURE2_HAS_ISEL
powerpc64: Check cacheline size before using optimised memset routines A number of optimised memset routines assume the cacheline size is 128B, so we better check before using them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:50 +00:00
&& hwcap & PPC_FEATURE_HAS_VSX
&& cacheline_size == 128,
powerpc64le: Optimize memset for POWER10 This implementation is based on __memset_power8 and integrates a lot of suggestions from Anton Blanchard. The biggest difference is that it makes extensive use of stxvl to alignment and tail code to avoid branches and small stores. It has three main execution paths: a) "Short lengths" for lengths up to 64 bytes, avoiding as many branches as possible. b) "General case" for larger lengths, it has an alignment section using stxvl to avoid branches, a 128 bytes loop and then a tail code, again using stxvl with few branches. c) "Zeroing cache blocks" for lengths from 256 bytes upwards and set value being zero. It is mostly the __memset_power8 code but the alignment phase was simplified because, at this point, address is already 16-bytes aligned and also changed to use vector stores. The tail code was also simplified to reuse the general case tail. All unaligned stores use stxvl instructions that do not generate alignment interrupts on POWER10, making it safe to use on caching-inhibited memory. On average, this implementation provides something around 30% improvement when compared to __memset_power8. Reviewed-by: Matheus Castanho <msc@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-04-30 21:12:08 +00:00
__memset_power10)
#endif
powerpc64: Check cacheline size before using optimised memset routines A number of optimised memset routines assume the cacheline size is 128B, so we better check before using them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:50 +00:00
IFUNC_IMPL_ADD (array, i, memset, hwcap2 & PPC_FEATURE2_ARCH_2_07
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
&& hwcap & PPC_FEATURE_HAS_ALTIVEC
powerpc64: Check cacheline size before using optimised memset routines A number of optimised memset routines assume the cacheline size is 128B, so we better check before using them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:50 +00:00
&& cacheline_size == 128,
PowerPC: memset optimization for POWER8/PPC64 This patch adds an optimized memset implementation for POWER8. For sizes from 0 to 255 bytes, a word/doubleword algorithm similar to POWER7 optimized one is used. For size higher than 255 two strategies are used: 1. If the constant is different than 0, the memory is written with altivec vector instruction; 2. If constant is 0, dbcz instructions are used. The loop is unrolled to clear 512 byte at time. Using vector instructions increases throughput considerable, with a double performance for sizes larger than 1024. The dcbz loops unrolls also shows performance improvement, by doubling throughput for sizes larger than 8192 bytes.
2014-07-15 16:19:09 +00:00
__memset_power8)
powerpc64: Check cacheline size before using optimised memset routines A number of optimised memset routines assume the cacheline size is 128B, so we better check before using them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:50 +00:00
IFUNC_IMPL_ADD (array, i, memset, hwcap & PPC_FEATURE_ARCH_2_06
&& cacheline_size == 128,
PowerPC: multiarch memset/bzero for PowerPC64
2013-12-13 19:33:16 +00:00
__memset_power7)
powerpc64: Check cacheline size before using optimised memset routines A number of optimised memset routines assume the cacheline size is 128B, so we better check before using them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:50 +00:00
IFUNC_IMPL_ADD (array, i, memset, hwcap & PPC_FEATURE_ARCH_2_05
&& cacheline_size == 128,
PowerPC: multiarch memset/bzero for PowerPC64
2013-12-13 19:33:16 +00:00
__memset_power6)
powerpc64: Check cacheline size before using optimised memset routines A number of optimised memset routines assume the cacheline size is 128B, so we better check before using them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:50 +00:00
IFUNC_IMPL_ADD (array, i, memset, hwcap & PPC_FEATURE_POWER4
&& cacheline_size == 128,
PowerPC: multiarch memset/bzero for PowerPC64
2013-12-13 19:33:16 +00:00
__memset_power4)
IFUNC_IMPL_ADD (array, i, memset, 1, __memset_ppc))
PowerPC: multiarch strlen for PowerPC64
2013-12-13 19:38:17 +00:00
PowerPC: multiarch strcpy for PowerPC64
2013-12-13 19:54:41 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strcpy.c. */
IFUNC_IMPL (i, name, strcpy,
powerpc: Optimized strcpy for POWER9 This version uses VSX store vector with length instructions and is significantly faster on small strings and relatively unaligned large strings, compared to the POWER8 version. A few examples: __strcpy_power9 __strcpy_power8 Length 16, alignments in bytes 0/ 0: 2.52454 4.62695 Length 412, alignments in bytes 4/ 0: 11.6 22.9185
2020-05-13 23:00:26 +00:00
#ifdef __LITTLE_ENDIAN__
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, strcpy, hwcap2 & PPC_FEATURE2_ARCH_3_00
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Optimized strcpy for POWER9 This version uses VSX store vector with length instructions and is significantly faster on small strings and relatively unaligned large strings, compared to the POWER8 version. A few examples: __strcpy_power9 __strcpy_power8 Length 16, alignments in bytes 0/ 0: 2.52454 4.62695 Length 412, alignments in bytes 4/ 0: 11.6 22.9185
2020-05-13 23:00:26 +00:00
__strcpy_power9)
#endif
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, strcpy, hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: Optimized st{r,p}cpy for POWER8/PPC64 This patch adds an optimized POWER8 strcpy using unaligned accesses. For strings up to 16 bytes the implementation first calculate the string size, like strlen, and issues a memcpy. For larger strings, source is first aligned to 16 bytes and then tested over a loop that reads 16 bytes am combine the cmpb results for speedup. Special case is added for page cross reads. It shows 30%-60% improvement over the optimized POWER7 one that uses only aligned accesses.
2014-12-23 11:59:44 +00:00
__strcpy_power8)
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, strcpy, hwcap & PPC_FEATURE_ARCH_2_06
&& hwcap & PPC_FEATURE_HAS_VSX,
PowerPC: multiarch strcpy for PowerPC64
2013-12-13 19:54:41 +00:00
__strcpy_power7)
IFUNC_IMPL_ADD (array, i, strcpy, 1,
__strcpy_ppc))
PowerPC: multiarch stpcpy for PowerPC64
2013-12-13 19:55:22 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/stpcpy.c. */
IFUNC_IMPL (i, name, stpcpy,
powerpc: Optimized stpcpy for POWER9 Add stpcpy support to the POWER9 strcpy. This is up to 40% faster on small strings and up to 90% faster on long relatively unaligned strings, compared to the POWER8 version. A few examples: __stpcpy_power9 __stpcpy_power8 Length 20, alignments in bytes 4/ 4: 2.58246 4.8788 Length 1024, alignments in bytes 1/ 6: 24.8186 47.8528
2020-05-13 23:08:35 +00:00
#ifdef __LITTLE_ENDIAN__
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, stpcpy, hwcap2 & PPC_FEATURE2_ARCH_3_00
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Optimized stpcpy for POWER9 Add stpcpy support to the POWER9 strcpy. This is up to 40% faster on small strings and up to 90% faster on long relatively unaligned strings, compared to the POWER8 version. A few examples: __stpcpy_power9 __stpcpy_power8 Length 20, alignments in bytes 4/ 4: 2.58246 4.8788 Length 1024, alignments in bytes 1/ 6: 24.8186 47.8528
2020-05-13 23:08:35 +00:00
__stpcpy_power9)
#endif
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, stpcpy, hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: Optimized st{r,p}cpy for POWER8/PPC64 This patch adds an optimized POWER8 strcpy using unaligned accesses. For strings up to 16 bytes the implementation first calculate the string size, like strlen, and issues a memcpy. For larger strings, source is first aligned to 16 bytes and then tested over a loop that reads 16 bytes am combine the cmpb results for speedup. Special case is added for page cross reads. It shows 30%-60% improvement over the optimized POWER7 one that uses only aligned accesses.
2014-12-23 11:59:44 +00:00
__stpcpy_power8)
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, stpcpy, hwcap & PPC_FEATURE_ARCH_2_06
&& hwcap & PPC_FEATURE_HAS_VSX,
PowerPC: multiarch stpcpy for PowerPC64
2013-12-13 19:55:22 +00:00
__stpcpy_power7)
IFUNC_IMPL_ADD (array, i, stpcpy, 1,
__stpcpy_ppc))
PowerPC: multiarch strlen for PowerPC64
2013-12-13 19:38:17 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strlen.c. */
IFUNC_IMPL (i, name, strlen,
powerpc64le: add optimized strlen for P9 This started as a trivial change to Anton's rawmemchr. I got carried away. This is a hybrid between P8's asympotically faster 64B checks with extremely efficient small string checks e.g <64B (and sometimes a little bit more depending on alignment). The second trick is to align to 64B by running a 48B checking loop 16B at a time until we naturally align to 64B (i.e checking 48/96/144 bytes/iteration based on the alignment after the first 5 comparisons). This allieviates the need to check page boundaries. Finally, explicly use the P7 strlen with the runtime loader when building P9. We need to be cautious about vector/vsx extensions here on P9 only builds.
2020-05-18 16:16:06 +00:00
#ifdef __LITTLE_ENDIAN__
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, strlen, hwcap2 & PPC_FEATURE2_ARCH_3_1
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Add optimized strlen for POWER10 Improvements compared to POWER9 version: 1. Take into account first 16B comparison for aligned strings The previous version compares the first 16B and increments r4 by the number of bytes until the address is 16B-aligned, then starts doing aligned loads at that address. For aligned strings, this causes the first 16B to be compared twice, because the increment is 0. Here we calculate the next 16B-aligned address differently, which avoids that issue. 2. Use simple comparisons for the first ~192 bytes The main loop is good for big strings, but comparing 16B each time is better for smaller strings. So after aligning the address to 16 Bytes, we check more 176B in 16B chunks. There may be some overlaps with the main loop for unaligned strings, but we avoid using the more aggressive strategy too soon, and also allow the loop to start at a 64B-aligned address. This greatly benefits smaller strings and avoids overlapping checks if the string is already aligned at a 64B boundary. 3. Reduce dependencies between load blocks caused by address calculation on loop Doing a precise time tracing on the code showed many loads in the loop were stalled waiting for updates to r4 from previous code blocks. This implementation avoids that as much as possible by using 2 registers (r4 and r5) to hold addresses to be used by different parts of the code. Also, the previous code aligned the address to 16B, then to 64B by doing a few 48B loops (if needed) until the address was aligned. The main loop could not start until that 48B loop had finished and r4 was updated with the current address. Here we calculate the address used by the loop very early, so it can start sooner. The main loop now uses 2 pointers 128B apart to make pointer updates less frequent, and also unrolls 1 iteration to guarantee there is enough time between iterations to update the pointers, reducing stalled cycles. 4. Use new P10 instructions lxvp is used to load 32B with a single instruction, reducing contention in the load queue. vextractbm allows simplifying the tail code for the loop, replacing vbpermq and avoiding having to generate a permute control vector. Reviewed-by: Paul E Murphy <murphyp@linux.ibm.com> Reviewed-by: Raphael M Zinsly <rzinsly@linux.ibm.com> Reviewed-by: Lucas A. M. Magalhaes <lamm@linux.ibm.com>
2020-09-29 18:40:08 +00:00
__strlen_power10)
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, strlen, hwcap2 & PPC_FEATURE2_ARCH_3_00
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc64le: add optimized strlen for P9 This started as a trivial change to Anton's rawmemchr. I got carried away. This is a hybrid between P8's asympotically faster 64B checks with extremely efficient small string checks e.g <64B (and sometimes a little bit more depending on alignment). The second trick is to align to 64B by running a 48B checking loop 16B at a time until we naturally align to 64B (i.e checking 48/96/144 bytes/iteration based on the alignment after the first 5 comparisons). This allieviates the need to check page boundaries. Finally, explicly use the P7 strlen with the runtime loader when building P9. We need to be cautious about vector/vsx extensions here on P9 only builds.
2020-05-18 16:16:06 +00:00
__strlen_power9)
#endif
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, strlen, hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: Optimization for strlen for POWER8. This implementation takes advantage of vectorization to improve performance of the loop over the current strlen implementation for POWER7.
2015-11-11 19:31:28 +00:00
__strlen_power8)
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
IFUNC_IMPL_ADD (array, i, strlen, hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strlen for PowerPC64
2013-12-13 19:38:17 +00:00
__strlen_power7)
IFUNC_IMPL_ADD (array, i, strlen, 1,
__strlen_ppc))
PowerPC: multiarch strncmp for PowerPC64
2013-12-13 19:48:48 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strncmp.c. */
IFUNC_IMPL (i, name, strncmp,
powerpc: Rearrange little endian specific files This patch moves little endian specific POWER9 optimization files to sysdeps/powerpc/powerpc64/le and creates POWER9 ifunc functions only for little endian.
2018-08-16 06:42:02 +00:00
#ifdef __LITTLE_ENDIAN__
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, strncmp, hwcap2 & PPC_FEATURE2_ARCH_3_00
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: strncmp optimization for power9 Vectorized loops are used for strings > 32B when compared to power8 optimization. Tested on power9 ppc64le simulator.
2016-12-13 05:23:42 +00:00
__strncmp_power9)
powerpc: Rearrange little endian specific files This patch moves little endian specific POWER9 optimization files to sysdeps/powerpc/powerpc64/le and creates POWER9 ifunc functions only for little endian.
2018-08-16 06:42:02 +00:00
#endif
powerpc: Optimized strncmp for POWER8/PPC64 This patch adds an optimized POWER8 strncmp. The implementation focus on speeding up unaligned cases follwing the ideas of power8 strcmp. The algorithm first check the initial 16 bytes, then align the first function source and uses unaligned loads on second argument only. Aditional checks for page boundaries are done for unaligned cases (where sources alignment are different).
2015-01-09 21:04:26 +00:00
IFUNC_IMPL_ADD (array, i, strncmp, hwcap2 & PPC_FEATURE2_ARCH_2_07,
__strncmp_power8)
PowerPC: multiarch strncmp for PowerPC64
2013-12-13 19:48:48 +00:00
IFUNC_IMPL_ADD (array, i, strncmp, 1,
__strncmp_ppc))
PowerPC: multiarch strchr for PowerPC64
2013-12-13 19:49:54 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strchr.c. */
IFUNC_IMPL (i, name, strchr,
powerpc64: strchr/strchrnul optimization for power8 The P7 code is used for <=32B strings and for > 32B vectorized loops are used. This shows as an average 25% improvement depending on the position of search character. The performance is same for shorter strings. Tested on ppc64 and ppc64le.
2016-12-27 19:48:37 +00:00
IFUNC_IMPL_ADD (array, i, strchr,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc64: strchr/strchrnul optimization for power8 The P7 code is used for <=32B strings and for > 32B vectorized loops are used. This shows as an average 25% improvement depending on the position of search character. The performance is same for shorter strings. Tested on ppc64 and ppc64le.
2016-12-27 19:48:37 +00:00
__strchr_power8)
PowerPC: multiarch strchr for PowerPC64
2013-12-13 19:49:54 +00:00
IFUNC_IMPL_ADD (array, i, strchr,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strchr for PowerPC64
2013-12-13 19:49:54 +00:00
__strchr_power7)
IFUNC_IMPL_ADD (array, i, strchr, 1,
__strchr_ppc))
PowerPC: multiarch strchrnul for PowerPC64
2013-12-13 19:50:26 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strchrnul.c. */
IFUNC_IMPL (i, name, strchrnul,
powerpc64: strchr/strchrnul optimization for power8 The P7 code is used for <=32B strings and for > 32B vectorized loops are used. This shows as an average 25% improvement depending on the position of search character. The performance is same for shorter strings. Tested on ppc64 and ppc64le.
2016-12-27 19:48:37 +00:00
IFUNC_IMPL_ADD (array, i, strchrnul,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc64: strchr/strchrnul optimization for power8 The P7 code is used for <=32B strings and for > 32B vectorized loops are used. This shows as an average 25% improvement depending on the position of search character. The performance is same for shorter strings. Tested on ppc64 and ppc64le.
2016-12-27 19:48:37 +00:00
__strchrnul_power8)
PowerPC: multiarch strchrnul for PowerPC64
2013-12-13 19:50:26 +00:00
IFUNC_IMPL_ADD (array, i, strchrnul,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strchrnul for PowerPC64
2013-12-13 19:50:26 +00:00
__strchrnul_power7)
IFUNC_IMPL_ADD (array, i, strchrnul, 1,
__strchrnul_ppc))
PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
#endif
PowerPC: multirach memcmp for PowerPC64
2013-12-13 19:32:31 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/memcmp.c. */
IFUNC_IMPL (i, name, memcmp,
powerpc: Optimized memcmp for power10 This patch was based on the __memcmp_power8 and the recent __strlen_power10. Improvements from __memcmp_power8: 1. Don't need alignment code. On POWER10 lxvp and lxvl do not generate alignment interrupts, so they are safe for use on caching-inhibited memory. Notice that the comparison on the main loop will wait for both VSR to be ready. Therefore aligning one of the input address does not improve performance. In order to align both registers a vperm is necessary which add too much overhead. 2. Uses new POWER10 instructions This code uses lxvp to decrease contention on load by loading 32 bytes per instruction. The vextractbm is used to have a smaller tail code for calculating the return value. 3. Performance improvement This version has around 35% better performance on average. I saw no performance regressions for any length or alignment. Thanks Matheus for helping me out with some details. Co-authored-by: Matheus Castanho <msc@linux.ibm.com> Reviewed-by: Raphael M Zinsly <rzinsly@linux.ibm.com>
2021-05-06 20:01:52 +00:00
#ifdef __LITTLE_ENDIAN__
IFUNC_IMPL_ADD (array, i, memcmp,
hwcap2 & PPC_FEATURE2_ARCH_3_1
&& hwcap & PPC_FEATURE_HAS_VSX,
__memcmp_power10)
#endif
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
IFUNC_IMPL_ADD (array, i, memcmp, hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: Improve memcmp performance for POWER8 Vectorization improves performance over the current implementation. Tested on powerpc64 and powerpc64le.
2017-05-18 05:51:20 +00:00
__memcmp_power8)
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
IFUNC_IMPL_ADD (array, i, memcmp, hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multirach memcmp for PowerPC64
2013-12-13 19:32:31 +00:00
__memcmp_power7)
IFUNC_IMPL_ADD (array, i, memcmp, hwcap & PPC_FEATURE_POWER4,
__memcmp_power4)
IFUNC_IMPL_ADD (array, i, memcmp, 1, __memcmp_ppc))
PowerPC: multiarch mempcpy for PowerPC64
2013-12-13 19:34:06 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/mempcpy.c. */
IFUNC_IMPL (i, name, mempcpy,
IFUNC_IMPL_ADD (array, i, mempcpy,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap & PPC_FEATURE_ARCH_2_06
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
PowerPC: multiarch mempcpy for PowerPC64
2013-12-13 19:34:06 +00:00
__mempcpy_power7)
IFUNC_IMPL_ADD (array, i, mempcpy, 1,
__mempcpy_ppc))
PowerPC: multiarch memchr for PowerPC64
2013-12-13 19:35:28 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/memchr.c. */
IFUNC_IMPL (i, name, memchr,
powerpc: Optimize memchr for power8 Vectorized loops are used for sizes greater than 32B to improve performance over power7 optimiztion.
2017-06-21 05:25:12 +00:00
IFUNC_IMPL_ADD (array, i, memchr,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: Optimize memchr for power8 Vectorized loops are used for sizes greater than 32B to improve performance over power7 optimiztion.
2017-06-21 05:25:12 +00:00
__memchr_power8)
PowerPC: multiarch memchr for PowerPC64
2013-12-13 19:35:28 +00:00
IFUNC_IMPL_ADD (array, i, memchr,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch memchr for PowerPC64
2013-12-13 19:35:28 +00:00
__memchr_power7)
IFUNC_IMPL_ADD (array, i, memchr, 1,
__memchr_ppc))
PowerPC: multiarch memrchr for PowerPC64
2013-12-13 19:36:50 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/memrchr.c. */
IFUNC_IMPL (i, name, memrchr,
powerpc: Optimize memrchr for power8 Vectorized loops are used for sizes greater than 32B to improve performance over power7 optimization. This shows as an average of 25% improvement depending on the position of search character. The performance is same for shorter strings.
2017-10-02 12:01:13 +00:00
IFUNC_IMPL_ADD (array, i, memrchr,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: Optimize memrchr for power8 Vectorized loops are used for sizes greater than 32B to improve performance over power7 optimization. This shows as an average of 25% improvement depending on the position of search character. The performance is same for shorter strings.
2017-10-02 12:01:13 +00:00
__memrchr_power8)
PowerPC: multiarch memrchr for PowerPC64
2013-12-13 19:36:50 +00:00
IFUNC_IMPL_ADD (array, i, memrchr,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch memrchr for PowerPC64
2013-12-13 19:36:50 +00:00
__memrchr_power7)
IFUNC_IMPL_ADD (array, i, memrchr, 1,
__memrchr_ppc))
PowerPC: multiarch rawmemchr for PowerPC64
2013-12-13 19:37:26 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/rawmemchr.c. */
IFUNC_IMPL (i, name, rawmemchr,
powerpc: Optimized rawmemchr for POWER9 This version uses vector instructions and is up to 60% faster on medium matches and up to 90% faster on long matches, compared to the POWER7 version. A few examples: __rawmemchr_power9 __rawmemchr_power7 Length 32, alignment 0: 2.27566 3.77765 Length 64, alignment 2: 2.46231 3.51064 Length 1024, alignment 0: 17.3059 32.6678
2020-05-14 11:49:16 +00:00
#ifdef __LITTLE_ENDIAN__
powerpc: Add optimized rawmemchr for POWER10 Reuse code for optimized strlen to implement a faster version of rawmemchr. This takes advantage of the same benefits provided by the strlen implementation, but needs some extra steps. __strlen_power10 code should be unchanged after this change. rawmemchr returns a pointer to the char found, while strlen returns only the length, so we have to take that into account when preparing the return value. To quickly check 64B, the loop on __strlen_power10 merges the whole block into 16B by using unsigned minimum vector operations (vminub) and checks if there are any \0 on the resulting vector. The same code is used by rawmemchr if the char c is 0. However, this approach does not work when c != 0. We first need to subtract each byte by c, so that the value we are looking for is converted to a 0, then taking the minimum and checking for nulls works again. The new code branches after it has compared ~256 bytes and chooses which of the two strategies above will be used in the main loop, based on the char c. This extra branch adds some overhead (~5%) for length ~256, but is quickly amortized by the faster loop for larger sizes. Compared to __rawmemchr_power9, this version is ~20% faster for length < 256. Because of the optimized main loop, the improvement becomes ~35% for c != 0 and ~50% for c = 0 for strings longer than 256. Reviewed-by: Lucas A. M. Magalhaes <lamm@linux.ibm.com> Reviewed-by: Raphael M Zinsly <rzinsly@linux.ibm.com>
2021-05-11 20:53:07 +00:00
IFUNC_IMPL_ADD (array, i, rawmemchr,
(hwcap2 & PPC_FEATURE2_ARCH_3_1)
&& (hwcap & PPC_FEATURE_HAS_VSX),
__rawmemchr_power10)
powerpc: Optimized rawmemchr for POWER9 This version uses vector instructions and is up to 60% faster on medium matches and up to 90% faster on long matches, compared to the POWER7 version. A few examples: __rawmemchr_power9 __rawmemchr_power7 Length 32, alignment 0: 2.27566 3.77765 Length 64, alignment 2: 2.46231 3.51064 Length 1024, alignment 0: 17.3059 32.6678
2020-05-14 11:49:16 +00:00
IFUNC_IMPL_ADD (array, i, rawmemchr,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_3_00
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Optimized rawmemchr for POWER9 This version uses vector instructions and is up to 60% faster on medium matches and up to 90% faster on long matches, compared to the POWER7 version. A few examples: __rawmemchr_power9 __rawmemchr_power7 Length 32, alignment 0: 2.27566 3.77765 Length 64, alignment 2: 2.46231 3.51064 Length 1024, alignment 0: 17.3059 32.6678
2020-05-14 11:49:16 +00:00
__rawmemchr_power9)
#endif
PowerPC: multiarch rawmemchr for PowerPC64
2013-12-13 19:37:26 +00:00
IFUNC_IMPL_ADD (array, i, rawmemchr,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch rawmemchr for PowerPC64
2013-12-13 19:37:26 +00:00
__rawmemchr_power7)
IFUNC_IMPL_ADD (array, i, rawmemchr, 1,
__rawmemchr_ppc))
PowerPC: multiarch strnlen for PowerPC64
2013-12-13 19:38:50 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strnlen.c. */
IFUNC_IMPL (i, name, strnlen,
powerpc64: Add POWER8 strnlen Added strnlen POWER8 otimized for long strings. It delivers same performance as POWER7 implementation for short strings. This takes advantage of reasonably performing unaligned loads and bit permutes to check the first 1-16 bytes until quadword aligned, then checks in 64 bytes strides until unsafe, then 16 bytes, truncating the count if need be. Likewise, the POWER7 code is recycled for less than 32 bytes strings. Tested on ppc64 and ppc64le. * sysdeps/powerpc/powerpc64/multiarch/Makefile (sysdep_routines): Add strnlen-power8. * sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c (strnlen): Add __strnlen_power8 to list of strnlen functions. * sysdeps/powerpc/powerpc64/multiarch/strnlen-power8.S: New file. * sysdeps/powerpc/powerpc64/multiarch/strnlen.c (__strnlen): Add __strnlen_power8 to ifunc list. * sysdeps/powerpc/powerpc64/power8/strnlen.S: New file.
2017-04-05 13:24:24 +00:00
IFUNC_IMPL_ADD (array, i, strnlen,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc64: Add POWER8 strnlen Added strnlen POWER8 otimized for long strings. It delivers same performance as POWER7 implementation for short strings. This takes advantage of reasonably performing unaligned loads and bit permutes to check the first 1-16 bytes until quadword aligned, then checks in 64 bytes strides until unsafe, then 16 bytes, truncating the count if need be. Likewise, the POWER7 code is recycled for less than 32 bytes strings. Tested on ppc64 and ppc64le. * sysdeps/powerpc/powerpc64/multiarch/Makefile (sysdep_routines): Add strnlen-power8. * sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c (strnlen): Add __strnlen_power8 to list of strnlen functions. * sysdeps/powerpc/powerpc64/multiarch/strnlen-power8.S: New file. * sysdeps/powerpc/powerpc64/multiarch/strnlen.c (__strnlen): Add __strnlen_power8 to ifunc list. * sysdeps/powerpc/powerpc64/power8/strnlen.S: New file.
2017-04-05 13:24:24 +00:00
__strnlen_power8)
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
IFUNC_IMPL_ADD (array, i, strnlen, hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strnlen for PowerPC64
2013-12-13 19:38:50 +00:00
__strnlen_power7)
IFUNC_IMPL_ADD (array, i, strnlen, 1,
__strnlen_ppc))
PowerPC: multiarch strcasecmp for PowerPC64
2013-12-13 19:39:51 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strcasecmp.c. */
IFUNC_IMPL (i, name, strcasecmp,
powerpc: strcasecmp/strncasecmp optmization for power8 This implementation utilizes vectors to improve performance compared to current byte by byte implementation for POWER7. The performance improvement is upto 4x. This patch is tested on powerpc64 and powerpc64le.
2016-06-14 09:21:16 +00:00
IFUNC_IMPL_ADD (array, i, strcasecmp,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: strcasecmp/strncasecmp optmization for power8 This implementation utilizes vectors to improve performance compared to current byte by byte implementation for POWER7. The performance improvement is upto 4x. This patch is tested on powerpc64 and powerpc64le.
2016-06-14 09:21:16 +00:00
__strcasecmp_power8)
PowerPC: multiarch strcasecmp for PowerPC64
2013-12-13 19:39:51 +00:00
IFUNC_IMPL_ADD (array, i, strcasecmp,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strcasecmp for PowerPC64
2013-12-13 19:39:51 +00:00
__strcasecmp_power7)
IFUNC_IMPL_ADD (array, i, strcasecmp, 1, __strcasecmp_ppc))
/* Support sysdeps/powerpc/powerpc64/multiarch/strcasecmp_l.c. */
IFUNC_IMPL (i, name, strcasecmp_l,
IFUNC_IMPL_ADD (array, i, strcasecmp_l,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strcasecmp for PowerPC64
2013-12-13 19:39:51 +00:00
__strcasecmp_l_power7)
IFUNC_IMPL_ADD (array, i, strcasecmp_l, 1,
__strcasecmp_l_ppc))
PowerPC: multiarch strncasecmp for PowerPC64
2013-12-13 19:40:28 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strncase.c. */
IFUNC_IMPL (i, name, strncasecmp,
powerpc: strcasecmp/strncasecmp optmization for power8 This implementation utilizes vectors to improve performance compared to current byte by byte implementation for POWER7. The performance improvement is upto 4x. This patch is tested on powerpc64 and powerpc64le.
2016-06-14 09:21:16 +00:00
IFUNC_IMPL_ADD (array, i, strncasecmp,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: strcasecmp/strncasecmp optmization for power8 This implementation utilizes vectors to improve performance compared to current byte by byte implementation for POWER7. The performance improvement is upto 4x. This patch is tested on powerpc64 and powerpc64le.
2016-06-14 09:21:16 +00:00
__strncasecmp_power8)
PowerPC: multiarch strncasecmp for PowerPC64
2013-12-13 19:40:28 +00:00
IFUNC_IMPL_ADD (array, i, strncasecmp,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strncasecmp for PowerPC64
2013-12-13 19:40:28 +00:00
__strncasecmp_power7)
IFUNC_IMPL_ADD (array, i, strncasecmp, 1, __strncasecmp_ppc))
/* Support sysdeps/powerpc/powerpc64/multiarch/strncase_l.c. */
IFUNC_IMPL (i, name, strncasecmp_l,
IFUNC_IMPL_ADD (array, i, strncasecmp_l,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: multiarch strncasecmp for PowerPC64
2013-12-13 19:40:28 +00:00
__strncasecmp_l_power7)
IFUNC_IMPL_ADD (array, i, strncasecmp_l, 1,
__strncasecmp_l_ppc))
PowerPC: strrchr optimization for POWER7/PPC64 This patch optimizes strrchr() for ppc64. It uses aligned memory access along with cmpb instruction and CPU prefetch to avoid cache misses for speed improvement.
2014-03-03 14:06:41 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strrchr.c. */
IFUNC_IMPL (i, name, strrchr,
powerpc64: strrchr optimization for power8 P7 code is used for <=32B strings and for > 32B vectorized loops are used. This shows as an average 25% improvement depending on the position of search character. The performance is same for shorter strings. Tested on ppc64 and ppc64le.
2017-04-18 05:58:56 +00:00
IFUNC_IMPL_ADD (array, i, strrchr,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc64: strrchr optimization for power8 P7 code is used for <=32B strings and for > 32B vectorized loops are used. This shows as an average 25% improvement depending on the position of search character. The performance is same for shorter strings. Tested on ppc64 and ppc64le.
2017-04-18 05:58:56 +00:00
__strrchr_power8)
PowerPC: strrchr optimization for POWER7/PPC64 This patch optimizes strrchr() for ppc64. It uses aligned memory access along with cmpb instruction and CPU prefetch to avoid cache misses for speed improvement.
2014-03-03 14:06:41 +00:00
IFUNC_IMPL_ADD (array, i, strrchr,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: strrchr optimization for POWER7/PPC64 This patch optimizes strrchr() for ppc64. It uses aligned memory access along with cmpb instruction and CPU prefetch to avoid cache misses for speed improvement.
2014-03-03 14:06:41 +00:00
__strrchr_power7)
IFUNC_IMPL_ADD (array, i, strrchr, 1,
__strrchr_ppc))
PowerPC: strncat optimization for PPC64 The optimization is achieved by following techniques: 1. Doubleword aligned memory access and compares using cmpb instruction. 2. Loop unrolling for byte load/store. 3. CPU pre-fetch to avoid cache miss.
2014-03-07 12:09:47 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strncat.c. */
IFUNC_IMPL (i, name, strncat,
powerpc: Optimized strncat for POWER8 With new optimized strnlen for POWER8 [1], this patch adds strncat for power8 to make use of optimized strlen and strnlen. This is faster than POWER7 current implementation for larger strings. Tested on powerpc64 and powerpc64le. [1] https://sourceware.org/ml/libc-alpha/2017-03/msg00491.html * sysdeps/powerpc/powerpc64/multiarch/Makefile (sysdep_routines): Add strncat-power8. * sysdeps/powerpc/powerpc64/multiarch/strncat.c (strncat): Add __strncat_power8 to ifunc list. * sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c (strncat): Add __strncat_power8 to list of strncat functions. * sysdeps/powerpc/powerpc64/multiarch/strncat-power8.c: New file.
2017-04-13 05:59:20 +00:00
IFUNC_IMPL_ADD (array, i, strncat,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Optimized strncat for POWER8 With new optimized strnlen for POWER8 [1], this patch adds strncat for power8 to make use of optimized strlen and strnlen. This is faster than POWER7 current implementation for larger strings. Tested on powerpc64 and powerpc64le. [1] https://sourceware.org/ml/libc-alpha/2017-03/msg00491.html * sysdeps/powerpc/powerpc64/multiarch/Makefile (sysdep_routines): Add strncat-power8. * sysdeps/powerpc/powerpc64/multiarch/strncat.c (strncat): Add __strncat_power8 to ifunc list. * sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c (strncat): Add __strncat_power8 to list of strncat functions. * sysdeps/powerpc/powerpc64/multiarch/strncat-power8.c: New file.
2017-04-13 05:59:20 +00:00
__strncat_power8)
PowerPC: strncat optimization for PPC64 The optimization is achieved by following techniques: 1. Doubleword aligned memory access and compares using cmpb instruction. 2. Loop unrolling for byte load/store. 3. CPU pre-fetch to avoid cache miss.
2014-03-07 12:09:47 +00:00
IFUNC_IMPL_ADD (array, i, strncat,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap & PPC_FEATURE_ARCH_2_06
&& hwcap & PPC_FEATURE_HAS_VSX,
PowerPC: strncat optimization for PPC64 The optimization is achieved by following techniques: 1. Doubleword aligned memory access and compares using cmpb instruction. 2. Loop unrolling for byte load/store. 3. CPU pre-fetch to avoid cache miss.
2014-03-07 12:09:47 +00:00
__strncat_power7)
IFUNC_IMPL_ADD (array, i, strncat, 1,
__strncat_ppc))
PowerPC: strncpy/stpncpy optimization for PPC64/POWER7 The optimization is achieved by following techniques: > data alignment [gain from aligned memory access on read/write] > POWER7 gains performance with loop unrolling/unwinding [gain by reduction of branch penalty]. > zero padding done by calling optimized memset
2014-05-06 00:10:45 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strncpy.c. */
IFUNC_IMPL (i, name, strncpy,
powerpc: Add optimized strncpy for POWER9 Similar to the strcpy P9 optimization, this version uses VSX to improve performance. Reviewed-by: Matheus Castanho <msc@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-11-12 16:12:24 +00:00
#ifdef __LITTLE_ENDIAN__
IFUNC_IMPL_ADD (array, i, strncpy,
(hwcap2 & PPC_FEATURE2_ARCH_3_00)
&& (hwcap & PPC_FEATURE_HAS_VSX),
__strncpy_power9)
#endif
powerpc: Optimized st{r,p}ncpy for POWER8/PPC64 This patch adds an optimized POWER8 st{r,p}ncpy using unaligned accesses. It shows 10%-80% improvement over the optimized POWER7 one that uses only aligned accesses, specially on unaligned inputs. The algorithm first read and check 16 bytes (if inputs do not cross a 4K page size). The it realign source to 16-bytes and issue a 16 bytes read and compare loop to speedup null byte checks for large strings. Also, different from POWER7 optimization, the null pad is done inline in the implementation using possible unaligned accesses, instead of realying on a memset call. Special case is added for page cross reads.
2014-12-31 16:47:41 +00:00
IFUNC_IMPL_ADD (array, i, strncpy,
hwcap2 & PPC_FEATURE2_ARCH_2_07,
__strncpy_power8)
PowerPC: strncpy/stpncpy optimization for PPC64/POWER7 The optimization is achieved by following techniques: > data alignment [gain from aligned memory access on read/write] > POWER7 gains performance with loop unrolling/unwinding [gain by reduction of branch penalty]. > zero padding done by calling optimized memset
2014-05-06 00:10:45 +00:00
IFUNC_IMPL_ADD (array, i, strncpy,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: strncpy/stpncpy optimization for PPC64/POWER7 The optimization is achieved by following techniques: > data alignment [gain from aligned memory access on read/write] > POWER7 gains performance with loop unrolling/unwinding [gain by reduction of branch penalty]. > zero padding done by calling optimized memset
2014-05-06 00:10:45 +00:00
__strncpy_power7)
IFUNC_IMPL_ADD (array, i, strncpy, 1,
__strncpy_ppc))
/* Support sysdeps/powerpc/powerpc64/multiarch/stpncpy.c. */
IFUNC_IMPL (i, name, stpncpy,
powerpc: Add optimized stpncpy for POWER9 Add stpncpy support into the POWER9 strncpy. Reviewed-by: Matheus Castanho <msc@linux.ibm.com> Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2020-11-12 16:12:24 +00:00
#ifdef __LITTLE_ENDIAN__
IFUNC_IMPL_ADD (array, i, stpncpy,
(hwcap2 & PPC_FEATURE2_ARCH_3_00)
&& (hwcap & PPC_FEATURE_HAS_VSX),
__stpncpy_power9)
#endif
powerpc: Optimized st{r,p}ncpy for POWER8/PPC64 This patch adds an optimized POWER8 st{r,p}ncpy using unaligned accesses. It shows 10%-80% improvement over the optimized POWER7 one that uses only aligned accesses, specially on unaligned inputs. The algorithm first read and check 16 bytes (if inputs do not cross a 4K page size). The it realign source to 16-bytes and issue a 16 bytes read and compare loop to speedup null byte checks for large strings. Also, different from POWER7 optimization, the null pad is done inline in the implementation using possible unaligned accesses, instead of realying on a memset call. Special case is added for page cross reads.
2014-12-31 16:47:41 +00:00
IFUNC_IMPL_ADD (array, i, stpncpy,
hwcap2 & PPC_FEATURE2_ARCH_2_07,
__stpncpy_power8)
PowerPC: strncpy/stpncpy optimization for PPC64/POWER7 The optimization is achieved by following techniques: > data alignment [gain from aligned memory access on read/write] > POWER7 gains performance with loop unrolling/unwinding [gain by reduction of branch penalty]. > zero padding done by calling optimized memset
2014-05-06 00:10:45 +00:00
IFUNC_IMPL_ADD (array, i, stpncpy,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: strncpy/stpncpy optimization for PPC64/POWER7 The optimization is achieved by following techniques: > data alignment [gain from aligned memory access on read/write] > POWER7 gains performance with loop unrolling/unwinding [gain by reduction of branch penalty]. > zero padding done by calling optimized memset
2014-05-06 00:10:45 +00:00
__stpncpy_power7)
IFUNC_IMPL_ADD (array, i, stpncpy, 1,
__stpncpy_ppc))
PowerPC: Optimized strcmp for PPC64/POWER7 Optimization is achieved on 8 byte aligned strings with double word comparison using cmpb instruction. On unaligned strings loop unrolling is applied for Power7 gain.
2014-06-06 12:56:07 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strcmp.c. */
IFUNC_IMPL (i, name, strcmp,
powerpc: Rearrange little endian specific files This patch moves little endian specific POWER9 optimization files to sysdeps/powerpc/powerpc64/le and creates POWER9 ifunc functions only for little endian.
2018-08-16 06:42:02 +00:00
#ifdef __LITTLE_ENDIAN__
powerpc: strcmp optimization for power9 Vectorized loops are used for strings > 32B when compared to power8 optimization. Tested on power9 ppc64le simulator.
2016-12-01 06:05:43 +00:00
IFUNC_IMPL_ADD (array, i, strcmp,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_3_00
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: strcmp optimization for power9 Vectorized loops are used for strings > 32B when compared to power8 optimization. Tested on power9 ppc64le simulator.
2016-12-01 06:05:43 +00:00
__strcmp_power9)
powerpc: Rearrange little endian specific files This patch moves little endian specific POWER9 optimization files to sysdeps/powerpc/powerpc64/le and creates POWER9 ifunc functions only for little endian.
2018-08-16 06:42:02 +00:00
#endif
powerpc: Optimized strcmp for POWER8/PPC64 This patch adds an optimized POWER8 strcmp using unaligned accesses. The algorithm first check the initial 16 bytes, then align the first function source and uses unaligned loads on second argument only. Aditional checks for page boundaries are done for unaligned cases
2015-01-07 12:18:30 +00:00
IFUNC_IMPL_ADD (array, i, strcmp,
hwcap2 & PPC_FEATURE2_ARCH_2_07,
__strcmp_power8)
PowerPC: Optimized strcmp for PPC64/POWER7 Optimization is achieved on 8 byte aligned strings with double word comparison using cmpb instruction. On unaligned strings loop unrolling is applied for Power7 gain.
2014-06-06 12:56:07 +00:00
IFUNC_IMPL_ADD (array, i, strcmp,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
PowerPC: Optimized strcmp for PPC64/POWER7 Optimization is achieved on 8 byte aligned strings with double word comparison using cmpb instruction. On unaligned strings loop unrolling is applied for Power7 gain.
2014-06-06 12:56:07 +00:00
__strcmp_power7)
IFUNC_IMPL_ADD (array, i, strcmp, 1,
__strcmp_ppc))
PowerPC: strcat optimization for PPC64/POWER7 This patch adds an ifunc power7 strcat symbol that uses the logic on sysdeps/powerpc/strcat.c but call power7 strlen/strcpy symbols instead of default ones.
2014-06-12 03:21:20 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strcat.c. */
IFUNC_IMPL (i, name, strcat,
powerpc: Optimized strcat for POWER8/PPC64 With new optimized strcpy for POWER8, this patch adds an optimized strcat which uses it along with default implementation at strings/.
2014-12-23 18:36:34 +00:00
IFUNC_IMPL_ADD (array, i, strcat,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Optimized strcat for POWER8/PPC64 With new optimized strcpy for POWER8, this patch adds an optimized strcat which uses it along with default implementation at strings/.
2014-12-23 18:36:34 +00:00
__strcat_power8)
PowerPC: strcat optimization for PPC64/POWER7 This patch adds an ifunc power7 strcat symbol that uses the logic on sysdeps/powerpc/strcat.c but call power7 strlen/strcpy symbols instead of default ones.
2014-06-12 03:21:20 +00:00
IFUNC_IMPL_ADD (array, i, strcat,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap & PPC_FEATURE_ARCH_2_06
&& hwcap & PPC_FEATURE_HAS_VSX,
PowerPC: strcat optimization for PPC64/POWER7 This patch adds an ifunc power7 strcat symbol that uses the logic on sysdeps/powerpc/strcat.c but call power7 strlen/strcpy symbols instead of default ones.
2014-06-12 03:21:20 +00:00
__strcat_power7)
IFUNC_IMPL_ADD (array, i, strcat, 1,
__strcat_ppc))
powerpc: Add optimized P8 strspn This utilizes vectors and bitmasks. For small needle, large haystack, the performance improvement is upto 8x. For short strings (0-4B), the cost of computing the bitmask dominates, and is a tad slower.
2016-03-14 21:40:46 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strspn.c. */
IFUNC_IMPL (i, name, strspn,
IFUNC_IMPL_ADD (array, i, strspn,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Add optimized P8 strspn This utilizes vectors and bitmasks. For small needle, large haystack, the performance improvement is upto 8x. For short strings (0-4B), the cost of computing the bitmask dominates, and is a tad slower.
2016-03-14 21:40:46 +00:00
__strspn_power8)
IFUNC_IMPL_ADD (array, i, strspn, 1,
__strspn_ppc))
powerpc: Add optimized strcspn for P8 A few minor adjustments to the P8 strspn gives us an almost equally optimized P8 strcspn.
2016-04-25 14:11:02 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strcspn.c. */
IFUNC_IMPL (i, name, strcspn,
IFUNC_IMPL_ADD (array, i, strcspn,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_VSX,
powerpc: Add optimized strcspn for P8 A few minor adjustments to the P8 strspn gives us an almost equally optimized P8 strcspn.
2016-04-25 14:11:02 +00:00
__strcspn_power8)
IFUNC_IMPL_ADD (array, i, strcspn, 1,
__strcspn_ppc))
powerpc: strstr optimization This patch optimizes strstr function for power >= 7 systems. Performance gain is obtained using aligned memory access and usage of cmpb instruction for quicker comparison. The average improvement of this optimization is ~40%. Tested on ppc64 and ppc64le. 2015-07-16 Rajalakshmi Srinivasaraghavan <raji@linux.vnet.ibm.com> * sysdeps/powerpc/powerpc64/multiarch/Makefile: Add strstr(). * sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c: Likewise. * sysdeps/powerpc/powerpc64/power7/strstr.S: New File. * sysdeps/powerpc/powerpc64/multiarch/strstr-power7.S: New File. * sysdeps/powerpc/powerpc64/multiarch/strstr-ppc64.c: New File. * sysdeps/powerpc/powerpc64/multiarch/strstr.c: New File.
2015-06-24 06:08:21 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strstr.c. */
IFUNC_IMPL (i, name, strstr,
IFUNC_IMPL_ADD (array, i, strstr,
powerpc64: Replace some PPC_FEATURE_HAS_VSX with PPC_FEATURE_ARCH_2_06 We use PPC_FEATURE_HAS_VSX to select a number of POWER7 optimised functions. These functions don't use any VSX instructions, so PPC_FEATURE_ARCH_2_06 seems like a better fit. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:49 +00:00
hwcap & PPC_FEATURE_ARCH_2_06,
powerpc: strstr optimization This patch optimizes strstr function for power >= 7 systems. Performance gain is obtained using aligned memory access and usage of cmpb instruction for quicker comparison. The average improvement of this optimization is ~40%. Tested on ppc64 and ppc64le. 2015-07-16 Rajalakshmi Srinivasaraghavan <raji@linux.vnet.ibm.com> * sysdeps/powerpc/powerpc64/multiarch/Makefile: Add strstr(). * sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c: Likewise. * sysdeps/powerpc/powerpc64/power7/strstr.S: New File. * sysdeps/powerpc/powerpc64/multiarch/strstr-power7.S: New File. * sysdeps/powerpc/powerpc64/multiarch/strstr-ppc64.c: New File. * sysdeps/powerpc/powerpc64/multiarch/strstr.c: New File.
2015-06-24 06:08:21 +00:00
__strstr_power7)
IFUNC_IMPL_ADD (array, i, strstr, 1,
__strstr_ppc))
powerpc: strcasestr optmization for power8 This patch optimizes strcasestr function for power >= 8 systems. The average improvement of this optimization is ~40% and compares 16 bytes at a time using vector instructions. This patch is tested on powerpc64 and powerpc64le.
2016-04-20 17:40:42 +00:00
/* Support sysdeps/powerpc/powerpc64/multiarch/strcasestr.c. */
IFUNC_IMPL (i, name, strcasestr,
IFUNC_IMPL_ADD (array, i, strcasestr,
powerpc64: Add checks for Altivec and VSX in ifunc selection We'd like to support processors without Altivec or VSX, so check the relevant hwcap bits before selecting them. Reviewed-by: Tulio Magno Quites Machado Filho <tuliom@linux.ibm.com>
2021-07-27 05:47:51 +00:00
hwcap2 & PPC_FEATURE2_ARCH_2_07
&& hwcap & PPC_FEATURE_HAS_ALTIVEC,
powerpc: strcasestr optmization for power8 This patch optimizes strcasestr function for power >= 8 systems. The average improvement of this optimization is ~40% and compares 16 bytes at a time using vector instructions. This patch is tested on powerpc64 and powerpc64le.
2016-04-20 17:40:42 +00:00
__strcasestr_power8)
IFUNC_IMPL_ADD (array, i, strcasestr, 1,
__strcasestr_ppc))
Add bounds check to __libc_ifunc_impl_list Add a proper bounds check to __libc_ifunc_impl_list. This makes MAX_IFUNC redundant and fixes several targets that will write outside the array. To avoid unnecessary large diffs, pass the maximum in the argument 'i' to IFUNC_IMPL_ADD - 'max' can be used in new ifunc definitions and existing ones can be updated if desired. Passes buildmanyglibc. Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
2022-06-10 16:13:29 +00:00
return 0;
PowerPC: multiarch memcpy for PowerPC64
2013-12-13 19:31:41 +00:00
}