JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into herbert/crypto-2.6.git
commit 8c5459f1663ee689f94e69b25adb415cb95acb88
Author: Zanussi, Tom <tom.zanussi@linux.intel.com>
Date: Fri Sep 27 13:46:49 2024 -0500
crypto: iaa - Remove potential infinite loop in check_completion()
For iaa_crypto operations, it's assumed that if an operation doesn't
make progress, the IAA watchdog timer will kick in and set the
completion status bit to failure and the reason to completion timeout.
Some systems may have broken hardware that doesn't even do that, which
can result in an infinite status-checking loop. Add a check for that
in the loop, and disable the driver if it occurs.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into the linux.git
commit e0d3b845a1b10b7b5abdad7ecc69d45b2aab3209
Author: Dan Carpenter <dan.carpenter@linaro.org>
Date: Wed Jul 24 11:09:43 2024 -0500
crypto: iaa - Fix potential use after free bug
The free_device_compression_mode(iaa_device, device_mode) function frees
"device_mode" but it iss passed to iaa_compression_modes[i]->free() a few
lines later resulting in a use after free.
The good news is that, so far as I can tell, nothing implements the
->free() function and the use after free happens in dead code. But, with
this fix, when something does implement it, we'll be ready. :)
Fixes: b190447e0fa3 ("crypto: iaa - Add compression mode management along with fixed mode")
Signed-off-by: Dan Carpenter <dan.carpenter@linaro.org>
Reviewed-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into the linux.git
commit bfbe27ba59e19e28801cc1a931a8f6d1d35b76d1
Author: Thorsten Blum <thorsten.blum@toblux.com>
Date: Thu May 2 17:33:39 2024 +0200
crypto: iaa - Use kmemdup() instead of kzalloc() and memcpy()
Fixes the following two Coccinelle/coccicheck warnings reported by
memdup.cocci:
iaa_crypto_main.c:350:19-26: WARNING opportunity for kmemdup
iaa_crypto_main.c:358:18-25: WARNING opportunity for kmemdup
Signed-off-by: Thorsten Blum <thorsten.blum@toblux.com>
Reviewed-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into the linux.git
commit 8f0e0cf74ccef41b383daddcf5447bba655031b3
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Fri Apr 5 13:57:30 2024 -0500
crypto: iaa - Use cpumask_weight() when rebalancing
If some cpus are offlined, or if the node mask is smaller than
expected, the 'nonexistent cpu' warning in rebalance_wq_table() may be
erroneously triggered.
Use cpumask_weight() to make sure we only iterate over the exact
number of cpus in the mask.
Also use num_possible_cpus() instead of num_online_cpus() to make sure
all slots in the wq table are initialized.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into the linux.git
commit 43698cd6c02ddcf3b4ffb34e5da0be3e801027fe
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Mon Mar 4 15:20:11 2024 -0600
crypto: iaa - Change iaa statistics to atomic64_t
Change all the iaa statistics to use atomic64_t instead of the current
u64, to avoid potentially inconsistent counts.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into the linux.git
commit c21fb22df63d51bb26d34023b0d9651a15442eb6
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Mon Mar 4 15:20:10 2024 -0600
crypto: iaa - Add global_stats file and remove individual stat files
Currently, the wq_stats output also includes the global stats, while
the individual global stats are also available as separate debugfs
files. Since these are all read-only, there's really no reason to
have them as separate files, especially since we already display them
as global stats in the wq_stats. It makes more sense to just add a
separate global_stats file to display those, and remove them from the
wq_stats, as well as removing the individual stats files.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into the linux.git
commit 956cb8a37039306379a1a926ccb1b55e08ffae80
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Mon Mar 4 15:20:09 2024 -0600
crypto: iaa - Remove comp/decomp delay statistics
As part of the simplification/cleanup of the iaa statistics, remove
the comp/decomp delay statistics.
They're actually not really useful and can be/are being more flexibly
generated using standard kernel tracing infrastructure.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-49539
Upstream Status: merged into the linux.git
commit 19b0ed5ddc8b554d1dfc34f870dc56e706ed205a
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Mon Mar 4 15:20:08 2024 -0600
crypto: iaa - fix decomp_bytes_in stats
Decomp stats should use slen, not dlen. Change both the global and
per-wq stats to use the correct value.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-29685
Upstream Status: RHEL only
The Intel Analytics Accelerator (IAA) Compression Accelerator driver
code has been merged upstream just recently. Taint the kernel with
TAINT_TECH_PREVIEW at the attempt to create the first workqueue.
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-29685
Upstream Status: merged into herbert/crypto-2.6.git
commit 5a7e89d3315d1be86aff8a8bf849023cda6547f7
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Thu Mar 21 16:08:45 2024 -0500
crypto: iaa - Fix nr_cpus < nr_iaa case
If nr_cpus < nr_iaa, the calculated cpus_per_iaa will be 0, which
causes a divide-by-0 in rebalance_wq_table().
Make sure cpus_per_iaa is 1 in that case, and also in the nr_iaa == 0
case, even though cpus_per_iaa is never used if nr_iaa == 0, for
paranoia.
Cc: <stable@vger.kernel.org> # v6.8+
Reported-by: Jerry Snitselaar <jsnitsel@redhat.com>
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-29685
Upstream Status: merged into the linux.git
commit 30dd94dba350043a32cfe9cb478ed621aae3c5c9
Author: Barry Song <v-songbaohua@oppo.com>
Date: Thu Feb 29 23:14:49 2024 +1300
crypto: iaa - fix the missing CRYPTO_ALG_ASYNC in cra_flags
Add the missing CRYPTO_ALG_ASYNC flag since intel iaa driver
works asynchronously.
Signed-off-by: Barry Song <v-songbaohua@oppo.com>
Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-29685
Upstream Status: merged into the linux.git
commit cdb083e73d632afcc5a931d31bb37445580f4bfb
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Sun Feb 25 14:11:34 2024 -0600
crypto: iaa - Fix comp/decomp delay statistics
The comp/decomp delay statistics currently have no callers; somehow
they were dropped during refactoring. There originally were also two
sets, one for the async algorithm, the other for the synchronous
version. Because the synchronous algorithm was dropped, one set should
be removed. To keep it consistent with the rest of the stats, and
since there's no ambiguity, remove the acomp/adecomp versions. Also
add back the callers.
Reported-by: Rex Zhang <rex.zhang@intel.com>
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-29685
Upstream Status: merged into the linux.git
commit 262534ddc88dfea7474ed18adfecf856e4fbe054
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Sun Feb 25 14:11:33 2024 -0600
crypto: iaa - Fix async_disable descriptor leak
The disable_async paths of iaa_compress/decompress() don't free idxd
descriptors in the async_disable case. Currently this only happens in
the testcases where req->dst is set to null. Add a test to free them
in those paths.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-29685
Upstream Status: merged into the linux.git
commit cc342dba0d39f226f4a5e26194404c3785481470
Author: Minjie Du <duminjie@vivo.com>
Date: Tue Jan 9 10:19:14 2024 +0800
crypto: iaa - Remove unnecessary debugfs_create_dir() error check in iaa_crypto_debugfs_init()
This patch removes the debugfs_create_dir() error checking in
iaa_crypto_debugfs_init(). Because the debugfs_create_dir() is developed
in a way that the caller can safely handle the errors that
occur during the creation of DebugFS nodes.
Signed-off-by: Minjie Du <duminjie@vivo.com>
Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-29685
Upstream Status: merged into the linux.git
commit 3274819b3c81c18c01e3c0e0ea726870ec237ac0
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Mon Jan 8 16:53:48 2024 -0600
crypto: iaa - Remove header table code
The header table and related code is currently unused - it was
included and used for canned mode, but canned mode has been removed,
so this code can be safely removed as well.
This indirectly fixes a bug reported by Dan Carpenter.
Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
Closes: https://lore.kernel.org/linux-crypto/b2e0bd974981291e16882686a2b9b1db3986abe4.camel@linux.intel.com/T/#m4403253d6a4347a925fab4fc1cdb4ef7c095fb86
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit b8910630c967ffee582289451ddb5f9f19c26872
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Wed Dec 27 14:28:32 2023 -0600
crypto: iaa - Account for cpu-less numa nodes
In some configurations e.g. systems with CXL, a numa node can have 0
cpus and cpumask_nth() will return a cpu value that doesn't exist,
which will result in an attempt to add an entry to the wq table at a
bad index.
To fix this, when iterating the cpus for a node, skip any node that
doesn't have cpus.
Also, as a precaution, add a warning and bail if cpumask_nth() returns
a nonexistent cpu.
Reported-by: Zhang, Rex <rex.zhang@intel.com>
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit 98bb0dd15133a66ce4d1079193367c536b9e3fec
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Mon Dec 18 14:47:14 2023 -0600
crypto: iaa - Change desc->priv to 0
In order for shared workqeues to work properly, desc->priv should be
set to 0 rather than 1. The need for this is described in commit
f5ccf55e1028 (dmaengine/idxd: Re-enable kernel workqueue under DMA
API), so we need to make IAA consistent with IOMMU settings, otherwise
we get:
[ 141.948389] IOMMU: dmar15: Page request in Privilege Mode
[ 141.948394] dmar15: Invalid page request: 2000026a100101 ffffb167
Dedicated workqueues ignore this field and are unaffected.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit 93382a91632a5d88bb9bb0ff1fea872fe87f5dc2
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Tue Dec 5 15:25:29 2023 -0600
crypto: iaa - Add IAA Compression Accelerator stats
Add support for optional debugfs statistics support for the IAA
Compression Accelerator. This is enabled by the kernel config item:
CRYPTO_DEV_IAA_CRYPTO_STATS
When enabled, the IAA crypto driver will generate statistics which can
be accessed at /sys/kernel/debug/iaa-crypto/.
See Documentation/driver-api/crypto/iax/iax-crypto.rst for details.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit 09646c98d0bfed47930d9eb0d66c323fae70a5e0
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Tue Dec 5 15:25:28 2023 -0600
crypto: iaa - Add irq support for the crypto async interface
The existing iaa crypto async support provides an implementation that
satisfies the interface but does so in a synchronous manner - it fills
and submits the IDXD descriptor and then waits for it to complete
before returning. This isn't a problem at the moment, since all
existing callers (e.g. zswap) wrap any asynchronous callees in a
synchronous wrapper anyway.
This change makes the iaa crypto async implementation truly
asynchronous: it fills and submits the IDXD descriptor, then returns
immediately with -EINPROGRESS. It also sets the descriptor's 'request
completion irq' bit and sets up a callback with the IDXD driver which
is called when the operation completes and the irq fires. The
existing callers such as zswap use synchronous wrappers to deal with
-EINPROGRESS and so work as expected without any changes.
This mode can be enabled by writing 'async_irq' to the sync_mode
iaa_crypto driver attribute:
echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode
Async mode without interrupts (caller must poll) can be enabled by
writing 'async' to it:
echo async > /sys/bus/dsa/drivers/crypto/sync_mode
The default sync mode can be enabled by writing 'sync' to it:
echo sync > /sys/bus/dsa/drivers/crypto/sync_mode
The sync_mode value setting at the time the IAA algorithms are
registered is captured in each algorithm's crypto_ctx and used for all
compresses and decompresses when using a given algorithm.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit 2ec6761df889fdf896fde761abd447596dd8f8c2
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Tue Dec 5 15:25:27 2023 -0600
crypto: iaa - Add support for deflate-iaa compression algorithm
This patch registers the deflate-iaa deflate compression algorithm and
hooks it up to the IAA hardware using the 'fixed' compression mode
introduced in the previous patch.
Because the IAA hardware has a 4k history-window limitation, only
buffers <= 4k, or that have been compressed using a <= 4k history
window, are technically compliant with the deflate spec, which allows
for a window of up to 32k. Because of this limitation, the IAA fixed
mode deflate algorithm is given its own algorithm name, 'deflate-iaa'.
With this change, the deflate-iaa crypto algorithm is registered and
operational, and compression and decompression operations are fully
enabled following the successful binding of the first IAA workqueue
to the iaa_crypto sub-driver.
when there are no IAA workqueues bound to the driver, the IAA crypto
algorithm can be unregistered by removing the module.
A new iaa_crypto 'verify_compress' driver attribute is also added,
allowing the user to toggle compression verification. If set, each
compress will be internally decompressed and the contents verified,
returning error codes if unsuccessful. This can be toggled with 0/1:
echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress
The default setting is '1' - verify all compresses.
The verify_compress value setting at the time the algorithm is
registered is captured in the algorithm's crypto_ctx and used for all
compresses when using the algorithm.
[ Based on work originally by George Powley, Jing Lin and Kyung Min
Park ]
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit b190447e0fa3ef7355480d641d078962e03768b4
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Tue Dec 5 15:25:26 2023 -0600
crypto: iaa - Add compression mode management along with fixed mode
Define an in-kernel API for adding and removing compression modes,
which can be used by kernel modules or other kernel code that
implements IAA compression modes.
Also add a separate file, iaa_crypto_comp_fixed.c, containing huffman
tables generated for the IAA 'fixed' compression mode. Future
compression modes can be added in a similar fashion.
One or more crypto compression algorithms will be created for each
compression mode, each of which can be selected as the compression
algorithm to be used by a particular facility.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit f57bf3f78377d66af89a6d0c6d926ffb1f590b5d
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Tue Dec 5 15:25:25 2023 -0600
crypto: iaa - Add per-cpu workqueue table with rebalancing
The iaa compression/decompression algorithms in later patches need a
way to retrieve an appropriate IAA workqueue depending on how close
the associated IAA device is to the current cpu.
For this purpose, add a per-cpu array of workqueues such that an
appropriate workqueue can be retrieved by simply accessing the per-cpu
array.
Whenever a new workqueue is bound to or unbound from the iaa_crypto
driver, the available workqueues are 'rebalanced' such that work
submitted from a particular CPU is given to the most appropriate
workqueue available. There currently isn't any way for the user to
tweak the way this is done internally - if necessary, knobs can be
added later for that purpose. Current best practice is to configure
and bind at least one workqueue for each IAA device, but as long as
there is at least one workqueue configured and bound to any IAA device
in the system, the iaa_crypto driver will work, albeit most likely not
as efficiently.
[ Based on work originally by George Powley, Jing Lin and Kyung Min
Park ]
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20145
Upstream Status: merged into the linux.git
commit ea7a5cbb43696cfacf73e61916d1860ac30b5b2f
Author: Tom Zanussi <tom.zanussi@linux.intel.com>
Date: Tue Dec 5 15:25:24 2023 -0600
crypto: iaa - Add Intel IAA Compression Accelerator crypto driver core
The Intel Analytics Accelerator (IAA) is a hardware accelerator that
provides very high thoughput compression/decompression compatible with
the DEFLATE compression standard described in RFC 1951, which is the
compression/decompression algorithm exported by this module.
Users can select IAA compress/decompress acceleration by specifying
one of the deflate-iaa* algorithms as the compression algorithm to use
by whatever facility allows asynchronous compression algorithms to be
selected.
For example, zswap can select the IAA fixed deflate algorithm
'deflate-iaa' via:
# echo deflate-iaa > /sys/module/zswap/parameters/compressor
This patch adds iaa_crypto as an idxd sub-driver and tracks iaa
devices and workqueues as they are probed or removed.
[ Based on work originally by George Powley, Jing Lin and Kyung Min
Park ]
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Vladis Dronov <vdronov@redhat.com>
Vladis Dronov