Centos-kernel-stream-9/drivers/crypto/intel/iaa/iaa_crypto.h

161 lines
3.5 KiB
C
Raw Normal View History

/* SPDX-License-Identifier: GPL-2.0 */
/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */
#ifndef __IAA_CRYPTO_H__
#define __IAA_CRYPTO_H__
#include <linux/crypto.h>
#include <linux/idxd.h>
#include <uapi/linux/idxd.h>
#define IDXD_SUBDRIVER_NAME "crypto"
crypto: iaa - Add support for deflate-iaa compression algorithm JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit 2ec6761df889fdf896fde761abd447596dd8f8c2 Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:27 2023 -0600 crypto: iaa - Add support for deflate-iaa compression algorithm This patch registers the deflate-iaa deflate compression algorithm and hooks it up to the IAA hardware using the 'fixed' compression mode introduced in the previous patch. Because the IAA hardware has a 4k history-window limitation, only buffers <= 4k, or that have been compressed using a <= 4k history window, are technically compliant with the deflate spec, which allows for a window of up to 32k. Because of this limitation, the IAA fixed mode deflate algorithm is given its own algorithm name, 'deflate-iaa'. With this change, the deflate-iaa crypto algorithm is registered and operational, and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver. when there are no IAA workqueues bound to the driver, the IAA crypto algorithm can be unregistered by removing the module. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. The verify_compress value setting at the time the algorithm is registered is captured in the algorithm's crypto_ctx and used for all compresses when using the algorithm. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
#define IAA_DECOMP_ENABLE BIT(0)
#define IAA_DECOMP_FLUSH_OUTPUT BIT(1)
#define IAA_DECOMP_CHECK_FOR_EOB BIT(2)
#define IAA_DECOMP_STOP_ON_EOB BIT(3)
#define IAA_DECOMP_SUPPRESS_OUTPUT BIT(9)
#define IAA_COMP_FLUSH_OUTPUT BIT(1)
#define IAA_COMP_APPEND_EOB BIT(2)
#define IAA_COMPLETION_TIMEOUT 1000000
#define IAA_ANALYTICS_ERROR 0x0a
#define IAA_ERROR_DECOMP_BUF_OVERFLOW 0x0b
#define IAA_ERROR_COMP_BUF_OVERFLOW 0x19
#define IAA_ERROR_WATCHDOG_EXPIRED 0x24
#define IAA_COMP_MODES_MAX 2
#define FIXED_HDR 0x2
#define FIXED_HDR_SIZE 3
crypto: iaa - Add support for deflate-iaa compression algorithm JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit 2ec6761df889fdf896fde761abd447596dd8f8c2 Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:27 2023 -0600 crypto: iaa - Add support for deflate-iaa compression algorithm This patch registers the deflate-iaa deflate compression algorithm and hooks it up to the IAA hardware using the 'fixed' compression mode introduced in the previous patch. Because the IAA hardware has a 4k history-window limitation, only buffers <= 4k, or that have been compressed using a <= 4k history window, are technically compliant with the deflate spec, which allows for a window of up to 32k. Because of this limitation, the IAA fixed mode deflate algorithm is given its own algorithm name, 'deflate-iaa'. With this change, the deflate-iaa crypto algorithm is registered and operational, and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver. when there are no IAA workqueues bound to the driver, the IAA crypto algorithm can be unregistered by removing the module. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. The verify_compress value setting at the time the algorithm is registered is captured in the algorithm's crypto_ctx and used for all compresses when using the algorithm. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
#define IAA_COMP_FLAGS (IAA_COMP_FLUSH_OUTPUT | \
IAA_COMP_APPEND_EOB)
#define IAA_DECOMP_FLAGS (IAA_DECOMP_ENABLE | \
IAA_DECOMP_FLUSH_OUTPUT | \
IAA_DECOMP_CHECK_FOR_EOB | \
IAA_DECOMP_STOP_ON_EOB)
/* Representation of IAA workqueue */
struct iaa_wq {
struct list_head list;
crypto: iaa - Add support for deflate-iaa compression algorithm JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit 2ec6761df889fdf896fde761abd447596dd8f8c2 Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:27 2023 -0600 crypto: iaa - Add support for deflate-iaa compression algorithm This patch registers the deflate-iaa deflate compression algorithm and hooks it up to the IAA hardware using the 'fixed' compression mode introduced in the previous patch. Because the IAA hardware has a 4k history-window limitation, only buffers <= 4k, or that have been compressed using a <= 4k history window, are technically compliant with the deflate spec, which allows for a window of up to 32k. Because of this limitation, the IAA fixed mode deflate algorithm is given its own algorithm name, 'deflate-iaa'. With this change, the deflate-iaa crypto algorithm is registered and operational, and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver. when there are no IAA workqueues bound to the driver, the IAA crypto algorithm can be unregistered by removing the module. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. The verify_compress value setting at the time the algorithm is registered is captured in the algorithm's crypto_ctx and used for all compresses when using the algorithm. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
struct idxd_wq *wq;
crypto: iaa - Add support for deflate-iaa compression algorithm JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit 2ec6761df889fdf896fde761abd447596dd8f8c2 Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:27 2023 -0600 crypto: iaa - Add support for deflate-iaa compression algorithm This patch registers the deflate-iaa deflate compression algorithm and hooks it up to the IAA hardware using the 'fixed' compression mode introduced in the previous patch. Because the IAA hardware has a 4k history-window limitation, only buffers <= 4k, or that have been compressed using a <= 4k history window, are technically compliant with the deflate spec, which allows for a window of up to 32k. Because of this limitation, the IAA fixed mode deflate algorithm is given its own algorithm name, 'deflate-iaa'. With this change, the deflate-iaa crypto algorithm is registered and operational, and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver. when there are no IAA workqueues bound to the driver, the IAA crypto algorithm can be unregistered by removing the module. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. The verify_compress value setting at the time the algorithm is registered is captured in the algorithm's crypto_ctx and used for all compresses when using the algorithm. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
int ref;
bool remove;
struct iaa_device *iaa_device;
};
struct iaa_device_compression_mode {
const char *name;
struct aecs_comp_table_record *aecs_comp_table;
struct aecs_decomp_table_record *aecs_decomp_table;
dma_addr_t aecs_comp_table_dma_addr;
dma_addr_t aecs_decomp_table_dma_addr;
};
/* Representation of IAA device with wqs, populated by probe */
struct iaa_device {
struct list_head list;
struct idxd_device *idxd;
struct iaa_device_compression_mode *compression_modes[IAA_COMP_MODES_MAX];
int n_wq;
struct list_head wqs;
};
crypto: iaa - Add per-cpu workqueue table with rebalancing JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit f57bf3f78377d66af89a6d0c6d926ffb1f590b5d Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:25 2023 -0600 crypto: iaa - Add per-cpu workqueue table with rebalancing The iaa compression/decompression algorithms in later patches need a way to retrieve an appropriate IAA workqueue depending on how close the associated IAA device is to the current cpu. For this purpose, add a per-cpu array of workqueues such that an appropriate workqueue can be retrieved by simply accessing the per-cpu array. Whenever a new workqueue is bound to or unbound from the iaa_crypto driver, the available workqueues are 'rebalanced' such that work submitted from a particular CPU is given to the most appropriate workqueue available. There currently isn't any way for the user to tweak the way this is done internally - if necessary, knobs can be added later for that purpose. Current best practice is to configure and bind at least one workqueue for each IAA device, but as long as there is at least one workqueue configured and bound to any IAA device in the system, the iaa_crypto driver will work, albeit most likely not as efficiently. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
struct wq_table_entry {
struct idxd_wq **wqs;
int max_wqs;
int n_wqs;
int cur_wq;
};
#define IAA_AECS_ALIGN 32
/*
* Analytics Engine Configuration and State (AECS) contains parameters and
* internal state of the analytics engine.
*/
struct aecs_comp_table_record {
u32 crc;
u32 xor_checksum;
u32 reserved0[5];
u32 num_output_accum_bits;
u8 output_accum[256];
u32 ll_sym[286];
u32 reserved1;
u32 reserved2;
u32 d_sym[30];
u32 reserved_padding[2];
} __packed;
/* AECS for decompress */
struct aecs_decomp_table_record {
u32 crc;
u32 xor_checksum;
u32 low_filter_param;
u32 high_filter_param;
u32 output_mod_idx;
u32 drop_init_decomp_out_bytes;
u32 reserved[36];
u32 output_accum_data[2];
u32 out_bits_valid;
u32 bit_off_indexing;
u32 input_accum_data[64];
u8 size_qw[32];
u32 decomp_state[1220];
} __packed;
int iaa_aecs_init_fixed(void);
void iaa_aecs_cleanup_fixed(void);
typedef int (*iaa_dev_comp_init_fn_t) (struct iaa_device_compression_mode *mode);
typedef int (*iaa_dev_comp_free_fn_t) (struct iaa_device_compression_mode *mode);
struct iaa_compression_mode {
const char *name;
u32 *ll_table;
int ll_table_size;
u32 *d_table;
int d_table_size;
u32 *header_table;
int header_table_size;
u16 gen_decomp_table_flags;
iaa_dev_comp_init_fn_t init;
iaa_dev_comp_free_fn_t free;
};
int add_iaa_compression_mode(const char *name,
const u32 *ll_table,
int ll_table_size,
const u32 *d_table,
int d_table_size,
const u8 *header_table,
int header_table_size,
u16 gen_decomp_table_flags,
iaa_dev_comp_init_fn_t init,
iaa_dev_comp_free_fn_t free);
void remove_iaa_compression_mode(const char *name);
crypto: iaa - Add support for deflate-iaa compression algorithm JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit 2ec6761df889fdf896fde761abd447596dd8f8c2 Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:27 2023 -0600 crypto: iaa - Add support for deflate-iaa compression algorithm This patch registers the deflate-iaa deflate compression algorithm and hooks it up to the IAA hardware using the 'fixed' compression mode introduced in the previous patch. Because the IAA hardware has a 4k history-window limitation, only buffers <= 4k, or that have been compressed using a <= 4k history window, are technically compliant with the deflate spec, which allows for a window of up to 32k. Because of this limitation, the IAA fixed mode deflate algorithm is given its own algorithm name, 'deflate-iaa'. With this change, the deflate-iaa crypto algorithm is registered and operational, and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver. when there are no IAA workqueues bound to the driver, the IAA crypto algorithm can be unregistered by removing the module. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. The verify_compress value setting at the time the algorithm is registered is captured in the algorithm's crypto_ctx and used for all compresses when using the algorithm. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
enum iaa_mode {
IAA_MODE_FIXED,
};
struct iaa_compression_ctx {
enum iaa_mode mode;
bool verify_compress;
crypto: iaa - Add irq support for the crypto async interface JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit 09646c98d0bfed47930d9eb0d66c323fae70a5e0 Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:28 2023 -0600 crypto: iaa - Add irq support for the crypto async interface The existing iaa crypto async support provides an implementation that satisfies the interface but does so in a synchronous manner - it fills and submits the IDXD descriptor and then waits for it to complete before returning. This isn't a problem at the moment, since all existing callers (e.g. zswap) wrap any asynchronous callees in a synchronous wrapper anyway. This change makes the iaa crypto async implementation truly asynchronous: it fills and submits the IDXD descriptor, then returns immediately with -EINPROGRESS. It also sets the descriptor's 'request completion irq' bit and sets up a callback with the IDXD driver which is called when the operation completes and the irq fires. The existing callers such as zswap use synchronous wrappers to deal with -EINPROGRESS and so work as expected without any changes. This mode can be enabled by writing 'async_irq' to the sync_mode iaa_crypto driver attribute: echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode Async mode without interrupts (caller must poll) can be enabled by writing 'async' to it: echo async > /sys/bus/dsa/drivers/crypto/sync_mode The default sync mode can be enabled by writing 'sync' to it: echo sync > /sys/bus/dsa/drivers/crypto/sync_mode The sync_mode value setting at the time the IAA algorithms are registered is captured in each algorithm's crypto_ctx and used for all compresses and decompresses when using a given algorithm. Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
bool async_mode;
bool use_irq;
crypto: iaa - Add support for deflate-iaa compression algorithm JIRA: https://issues.redhat.com/browse/RHEL-20145 Upstream Status: merged into the linux.git commit 2ec6761df889fdf896fde761abd447596dd8f8c2 Author: Tom Zanussi <tom.zanussi@linux.intel.com> Date: Tue Dec 5 15:25:27 2023 -0600 crypto: iaa - Add support for deflate-iaa compression algorithm This patch registers the deflate-iaa deflate compression algorithm and hooks it up to the IAA hardware using the 'fixed' compression mode introduced in the previous patch. Because the IAA hardware has a 4k history-window limitation, only buffers <= 4k, or that have been compressed using a <= 4k history window, are technically compliant with the deflate spec, which allows for a window of up to 32k. Because of this limitation, the IAA fixed mode deflate algorithm is given its own algorithm name, 'deflate-iaa'. With this change, the deflate-iaa crypto algorithm is registered and operational, and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver. when there are no IAA workqueues bound to the driver, the IAA crypto algorithm can be unregistered by removing the module. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. The verify_compress value setting at the time the algorithm is registered is captured in the algorithm's crypto_ctx and used for all compresses when using the algorithm. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Vladis Dronov <vdronov@redhat.com>
2024-01-17 14:29:12 +00:00
};
#endif