2020-03-21 11:19:07 +00:00
|
|
|
/* SPDX-License-Identifier: (GPL-2.0 WITH Linux-syscall-note) OR MIT */
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
/*
|
|
|
|
|
* Header file for the io_uring interface.
|
|
|
|
|
*
|
|
|
|
|
* Copyright (C) 2019 Jens Axboe
|
|
|
|
|
* Copyright (C) 2019 Christoph Hellwig
|
|
|
|
|
*/
|
|
|
|
|
#ifndef LINUX_IO_URING_H
|
|
|
|
|
#define LINUX_IO_URING_H
|
|
|
|
|
|
|
|
|
|
#include <linux/fs.h>
|
|
|
|
|
#include <linux/types.h>
|
2022-11-16 20:25:24 +00:00
|
|
|
/*
|
|
|
|
|
* this file is shared with liburing and that has to autodetect
|
|
|
|
|
* if linux/time_types.h is available or not, it can
|
|
|
|
|
* define UAPI_LINUX_IO_URING_H_SKIP_LINUX_TIME_TYPES_H
|
|
|
|
|
* if linux/time_types.h is not available
|
|
|
|
|
*/
|
|
|
|
|
#ifndef UAPI_LINUX_IO_URING_H_SKIP_LINUX_TIME_TYPES_H
|
io_uring: add sync cancelation API through io_uring_register()
The io_uring cancelation API is async, like any other API that we expose
there. For the case of finding a request to cancel, or not finding one,
it is fully sync in that when submission returns, the CQE for both the
cancelation request and the targeted request have been posted to the
CQ ring.
However, if the targeted work is being executed by io-wq, the API can
only start the act of canceling it. This makes it difficult to use in
some circumstances, as the caller then has to wait for the CQEs to come
in and match on the same cancelation data there.
Provide a IORING_REGISTER_SYNC_CANCEL command for io_uring_register()
that does sync cancelations, always. For the io-wq case, it'll wait
for the cancelation to come in before returning. The only expected
returns from this API is:
0 Request found and canceled fine.
> 0 Requests found and canceled. Only happens if asked to
cancel multiple requests, and if the work wasn't in
progress.
-ENOENT Request not found.
-ETIME A timeout on the operation was requested, but the timeout
expired before we could cancel.
and we won't get -EALREADY via this API.
If the timeout value passed in is -1 (tv_sec and tv_nsec), then that
means that no timeout is requested. Otherwise, the timespec passed in
is the amount of time the sync cancel will wait for a successful
cancelation.
Link: https://github.com/axboe/liburing/discussions/608
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-18 16:00:50 +00:00
|
|
|
#include <linux/time_types.h>
|
2022-11-16 20:25:24 +00:00
|
|
|
#endif
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
|
2022-08-23 11:45:49 +00:00
|
|
|
#ifdef __cplusplus
|
|
|
|
|
extern "C" {
|
|
|
|
|
#endif
|
|
|
|
|
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
/*
|
|
|
|
|
* IO submission data structure (Submission Queue Entry)
|
|
|
|
|
*/
|
|
|
|
|
struct io_uring_sqe {
|
|
|
|
|
__u8 opcode; /* type of operation for this sqe */
|
2019-01-11 05:13:58 +00:00
|
|
|
__u8 flags; /* IOSQE_ flags */
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
__u16 ioprio; /* ioprio for the request */
|
|
|
|
|
__s32 fd; /* file descriptor to do IO on */
|
2019-10-17 20:42:58 +00:00
|
|
|
union {
|
|
|
|
|
__u64 off; /* offset into file */
|
|
|
|
|
__u64 addr2;
|
2022-07-07 14:00:38 +00:00
|
|
|
struct {
|
|
|
|
|
__u32 cmd_op;
|
|
|
|
|
__u32 __pad1;
|
|
|
|
|
};
|
2019-10-17 20:42:58 +00:00
|
|
|
};
|
2020-02-24 08:32:45 +00:00
|
|
|
union {
|
|
|
|
|
__u64 addr; /* pointer to buffer or iovecs */
|
|
|
|
|
__u64 splice_off_in;
|
2023-10-16 13:47:47 +00:00
|
|
|
struct {
|
|
|
|
|
__u32 level;
|
|
|
|
|
__u32 optname;
|
|
|
|
|
};
|
2020-02-24 08:32:45 +00:00
|
|
|
};
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
__u32 len; /* buffer size or number of iovecs */
|
|
|
|
|
union {
|
|
|
|
|
__kernel_rwf_t rw_flags;
|
2019-01-11 16:43:02 +00:00
|
|
|
__u32 fsync_flags;
|
2020-06-17 09:53:55 +00:00
|
|
|
__u16 poll_events; /* compatibility */
|
|
|
|
|
__u32 poll32_events; /* word-reversed for BE */
|
2019-04-09 20:56:44 +00:00
|
|
|
__u32 sync_range_flags;
|
2019-04-19 19:34:07 +00:00
|
|
|
__u32 msg_flags;
|
2019-09-17 18:26:57 +00:00
|
|
|
__u32 timeout_flags;
|
2019-10-17 20:42:58 +00:00
|
|
|
__u32 accept_flags;
|
2019-10-29 03:49:21 +00:00
|
|
|
__u32 cancel_flags;
|
2019-12-11 18:20:36 +00:00
|
|
|
__u32 open_flags;
|
2019-12-14 04:18:10 +00:00
|
|
|
__u32 statx_flags;
|
2019-12-26 05:03:45 +00:00
|
|
|
__u32 fadvise_advice;
|
2020-02-24 08:32:45 +00:00
|
|
|
__u32 splice_flags;
|
2020-09-28 20:23:58 +00:00
|
|
|
__u32 rename_flags;
|
2020-09-28 20:27:37 +00:00
|
|
|
__u32 unlink_flags;
|
2021-07-08 06:34:47 +00:00
|
|
|
__u32 hardlink_flags;
|
2022-03-23 15:44:19 +00:00
|
|
|
__u32 xattr_flags;
|
io_uring: add support for passing fixed file descriptors
With IORING_OP_MSG_RING, one ring can send a message to another ring.
Extend that support to also allow sending a fixed file descriptor to
that ring, enabling one ring to pass a registered descriptor to another
one.
Arguments are extended to pass in:
sqe->addr3 fixed file slot in source ring
sqe->file_index fixed file slot in destination ring
IORING_OP_MSG_RING is extended to take a command argument in sqe->addr.
If set to zero (or IORING_MSG_DATA), it sends just a message like before.
If set to IORING_MSG_SEND_FD, a fixed file descriptor is sent according
to the above arguments.
Two common use cases for this are:
1) Server needs to be shutdown or restarted, pass file descriptors to
another onei
2) Backend is split, and one accepts connections, while others then get
the fd passed and handle the actual connection.
Both of those are classic SCM_RIGHTS use cases, and it's not possible to
support them with direct descriptors today.
By default, this will post a CQE to the target ring, similarly to how
IORING_MSG_DATA does it. If IORING_MSG_RING_CQE_SKIP is set, no message
is posted to the target ring. The issuer is expected to notify the
receiver side separately.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-13 10:47:02 +00:00
|
|
|
__u32 msg_ring_flags;
|
2022-09-30 06:27:39 +00:00
|
|
|
__u32 uring_cmd_flags;
|
2023-07-10 22:14:37 +00:00
|
|
|
__u32 waitid_flags;
|
io_uring: add support for futex wake and wait
Add support for FUTEX_WAKE/WAIT primitives.
IORING_OP_FUTEX_WAKE is mix of FUTEX_WAKE and FUTEX_WAKE_BITSET, as
it does support passing in a bitset.
Similary, IORING_OP_FUTEX_WAIT is a mix of FUTEX_WAIT and
FUTEX_WAIT_BITSET.
For both of them, they are using the futex2 interface.
FUTEX_WAKE is straight forward, as those can always be done directly from
the io_uring submission without needing async handling. For FUTEX_WAIT,
things are a bit more complicated. If the futex isn't ready, then we
rely on a callback via futex_queue->wake() when someone wakes up the
futex. From that calback, we queue up task_work with the original task,
which will post a CQE and wake it, if necessary.
Cancelations are supported, both from the application point-of-view,
but also to be able to cancel pending waits if the ring exits before
all events have occurred. The return value of futex_unqueue() is used
to gate who wins the potential race between cancelation and futex
wakeups. Whomever gets a 'ret == 1' return from that claims ownership
of the io_uring futex request.
This is just the barebones wait/wake support. PI or REQUEUE support is
not added at this point, unclear if we might look into that later.
Likewise, explicit timeouts are not supported either. It is expected
that users that need timeouts would do so via the usual io_uring
mechanism to do that using linked timeouts.
The SQE format is as follows:
`addr` Address of futex
`fd` futex2(2) FUTEX2_* flags
`futex_flags` io_uring specific command flags. None valid now.
`addr2` Value of futex
`addr3` Mask to wake/wait
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-06-08 17:57:40 +00:00
|
|
|
__u32 futex_flags;
|
2023-12-08 03:06:02 +00:00
|
|
|
__u32 install_fd_flags;
|
2024-05-10 03:50:28 +00:00
|
|
|
__u32 nop_flags;
|
2025-04-04 20:50:59 +00:00
|
|
|
__u32 pipe_flags;
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
};
|
|
|
|
|
__u64 user_data; /* data to be passed back at completion time */
|
2021-06-24 14:09:59 +00:00
|
|
|
/* pack this to avoid bogus arm OABI complaints */
|
io_uring: add support for pre-mapped user IO buffers
If we have fixed user buffers, we can map them into the kernel when we
setup the io_uring. That avoids the need to do get_user_pages() for
each and every IO.
To utilize this feature, the application must call io_uring_register()
after having setup an io_uring instance, passing in
IORING_REGISTER_BUFFERS as the opcode. The argument must be a pointer to
an iovec array, and the nr_args should contain how many iovecs the
application wishes to map.
If successful, these buffers are now mapped into the kernel, eligible
for IO. To use these fixed buffers, the application must use the
IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED opcodes, and then
set sqe->index to the desired buffer index. sqe->addr..sqe->addr+seq->len
must point to somewhere inside the indexed buffer.
The application may register buffers throughout the lifetime of the
io_uring instance. It can call io_uring_register() with
IORING_UNREGISTER_BUFFERS as the opcode to unregister the current set of
buffers, and then register a new set. The application need not
unregister buffers explicitly before shutting down the io_uring
instance.
It's perfectly valid to setup a larger buffer, and then sometimes only
use parts of it for an IO. As long as the range is within the originally
mapped region, it will work just fine.
For now, buffers must not be file backed. If file backed buffers are
passed in, the registration will fail with -1/EOPNOTSUPP. This
restriction may be relaxed in the future.
RLIMIT_MEMLOCK is used to check how much memory we can pin. A somewhat
arbitrary 1G per buffer size is also imposed.
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-09 16:16:05 +00:00
|
|
|
union {
|
2021-06-24 14:09:59 +00:00
|
|
|
/* index into fixed buffers, if used */
|
|
|
|
|
__u16 buf_index;
|
|
|
|
|
/* for grouped buffer selection */
|
|
|
|
|
__u16 buf_group;
|
|
|
|
|
} __attribute__((packed));
|
|
|
|
|
/* personality to use, if used */
|
|
|
|
|
__u16 personality;
|
2021-08-25 11:25:45 +00:00
|
|
|
union {
|
|
|
|
|
__s32 splice_fd_in;
|
|
|
|
|
__u32 file_index;
|
2025-02-15 00:09:41 +00:00
|
|
|
__u32 zcrx_ifq_idx;
|
2023-10-16 13:47:47 +00:00
|
|
|
__u32 optlen;
|
2022-07-12 20:52:43 +00:00
|
|
|
struct {
|
2022-07-12 20:52:45 +00:00
|
|
|
__u16 addr_len;
|
2022-09-01 10:54:04 +00:00
|
|
|
__u16 __pad3[1];
|
2022-07-12 20:52:43 +00:00
|
|
|
};
|
2025-05-06 12:17:27 +00:00
|
|
|
struct {
|
|
|
|
|
__u8 write_stream;
|
|
|
|
|
__u8 __pad4[3];
|
|
|
|
|
};
|
2021-08-25 11:25:45 +00:00
|
|
|
};
|
2022-05-11 05:47:45 +00:00
|
|
|
union {
|
|
|
|
|
struct {
|
|
|
|
|
__u64 addr3;
|
|
|
|
|
__u64 __pad2[1];
|
|
|
|
|
};
|
2024-11-28 11:22:36 +00:00
|
|
|
struct {
|
|
|
|
|
__u64 attr_ptr; /* pointer to attribute information */
|
|
|
|
|
__u64 attr_type_mask; /* bit mask of attributes */
|
|
|
|
|
};
|
2023-10-16 13:47:47 +00:00
|
|
|
__u64 optval;
|
2022-05-11 05:47:45 +00:00
|
|
|
/*
|
|
|
|
|
* If the ring is initialized with IORING_SETUP_SQE128, then
|
|
|
|
|
* this field is used for 80 bytes of arbitrary command data
|
|
|
|
|
*/
|
|
|
|
|
__u8 cmd[0];
|
|
|
|
|
};
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
};
|
|
|
|
|
|
2024-11-28 11:22:36 +00:00
|
|
|
/* sqe->attr_type_mask flags */
|
|
|
|
|
#define IORING_RW_ATTR_FLAG_PI (1U << 0)
|
|
|
|
|
/* PI attribute information */
|
|
|
|
|
struct io_uring_attr_pi {
|
|
|
|
|
__u16 flags;
|
|
|
|
|
__u16 app_tag;
|
|
|
|
|
__u32 len;
|
|
|
|
|
__u64 addr;
|
|
|
|
|
__u64 seed;
|
|
|
|
|
__u64 rsvd;
|
|
|
|
|
};
|
|
|
|
|
|
2022-05-07 20:18:44 +00:00
|
|
|
/*
|
|
|
|
|
* If sqe->file_index is set to this for opcodes that instantiate a new
|
|
|
|
|
* direct descriptor (like openat/openat2/accept), then io_uring will allocate
|
|
|
|
|
* an available direct descriptor instead of having the application pass one
|
|
|
|
|
* in. The picked direct descriptor will be returned in cqe->res, or -ENFILE
|
|
|
|
|
* if the space is full.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_FILE_INDEX_ALLOC (~0U)
|
|
|
|
|
|
2024-03-28 21:09:35 +00:00
|
|
|
enum io_uring_sqe_flags_bit {
|
2020-01-18 17:22:41 +00:00
|
|
|
IOSQE_FIXED_FILE_BIT,
|
|
|
|
|
IOSQE_IO_DRAIN_BIT,
|
|
|
|
|
IOSQE_IO_LINK_BIT,
|
|
|
|
|
IOSQE_IO_HARDLINK_BIT,
|
|
|
|
|
IOSQE_ASYNC_BIT,
|
io_uring: support buffer selection for OP_READ and OP_RECV
If a server process has tons of pending socket connections, generally
it uses epoll to wait for activity. When the socket is ready for reading
(or writing), the task can select a buffer and issue a recv/send on the
given fd.
Now that we have fast (non-async thread) support, a task can have tons
of pending reads or writes pending. But that means they need buffers to
back that data, and if the number of connections is high enough, having
them preallocated for all possible connections is unfeasible.
With IORING_OP_PROVIDE_BUFFERS, an application can register buffers to
use for any request. The request then sets IOSQE_BUFFER_SELECT in the
sqe, and a given group ID in sqe->buf_group. When the fd becomes ready,
a free buffer from the specified group is selected. If none are
available, the request is terminated with -ENOBUFS. If successful, the
CQE on completion will contain the buffer ID chosen in the cqe->flags
member, encoded as:
(buffer_id << IORING_CQE_BUFFER_SHIFT) | IORING_CQE_F_BUFFER;
Once a buffer has been consumed by a request, it is no longer available
and must be registered again with IORING_OP_PROVIDE_BUFFERS.
Requests need to support this feature. For now, IORING_OP_READ and
IORING_OP_RECV support it. This is checked on SQE submission, a CQE with
res == -EOPNOTSUPP will be posted if attempted on unsupported requests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-23 23:42:51 +00:00
|
|
|
IOSQE_BUFFER_SELECT_BIT,
|
io_uring: add option to skip CQE posting
Emitting a CQE is expensive from the kernel perspective. Often, it's
also not convenient for the userspace, spends some cycles on processing
and just complicates the logic. A similar problems goes for linked
requests, where we post an CQE for each request in the link.
Introduce a new flags, IOSQE_CQE_SKIP_SUCCESS, trying to help with it.
When set and a request completed successfully, it won't generate a CQE.
When fails, it produces an CQE, but all following linked requests will
be CQE-less, regardless whether they have IOSQE_CQE_SKIP_SUCCESS or not.
The notion of "fail" is the same as for link failing-cancellation, where
it's opcode dependent, and _usually_ result >= 0 is a success, but not
always.
Linked timeouts are a bit special. When the requests it's linked to was
not attempted to be executed, e.g. failing linked requests, it follows
the description above. Otherwise, whether a linked timeout will post a
completion or not solely depends on IOSQE_CQE_SKIP_SUCCESS of that
linked timeout request. Linked timeout never "fail" during execution, so
for them it's unconditional. It's expected for users to not really care
about the result of it but rely solely on the result of the master
request. Another reason for such a treatment is that it's racy, and the
timeout callback may be running awhile the master request posts its
completion.
use case 1:
If one doesn't care about results of some requests, e.g. normal
timeouts, just set IOSQE_CQE_SKIP_SUCCESS. Error result will still be
posted and need to be handled.
use case 2:
Set IOSQE_CQE_SKIP_SUCCESS for all requests of a link but the last,
and it'll post a completion only for the last one if everything goes
right, otherwise there will be one only one CQE for the first failed
request.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/0220fbe06f7cf99e6fc71b4297bb1cb6c0e89c2c.1636559119.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-11-10 15:49:32 +00:00
|
|
|
IOSQE_CQE_SKIP_SUCCESS_BIT,
|
2020-01-18 17:22:41 +00:00
|
|
|
};
|
|
|
|
|
|
2019-01-11 05:13:58 +00:00
|
|
|
/*
|
|
|
|
|
* sqe->flags
|
|
|
|
|
*/
|
2020-01-18 17:22:41 +00:00
|
|
|
/* use fixed fileset */
|
|
|
|
|
#define IOSQE_FIXED_FILE (1U << IOSQE_FIXED_FILE_BIT)
|
|
|
|
|
/* issue after inflight IO */
|
|
|
|
|
#define IOSQE_IO_DRAIN (1U << IOSQE_IO_DRAIN_BIT)
|
|
|
|
|
/* links next sqe */
|
|
|
|
|
#define IOSQE_IO_LINK (1U << IOSQE_IO_LINK_BIT)
|
|
|
|
|
/* like LINK, but stronger */
|
|
|
|
|
#define IOSQE_IO_HARDLINK (1U << IOSQE_IO_HARDLINK_BIT)
|
|
|
|
|
/* always go async */
|
|
|
|
|
#define IOSQE_ASYNC (1U << IOSQE_ASYNC_BIT)
|
io_uring: support buffer selection for OP_READ and OP_RECV
If a server process has tons of pending socket connections, generally
it uses epoll to wait for activity. When the socket is ready for reading
(or writing), the task can select a buffer and issue a recv/send on the
given fd.
Now that we have fast (non-async thread) support, a task can have tons
of pending reads or writes pending. But that means they need buffers to
back that data, and if the number of connections is high enough, having
them preallocated for all possible connections is unfeasible.
With IORING_OP_PROVIDE_BUFFERS, an application can register buffers to
use for any request. The request then sets IOSQE_BUFFER_SELECT in the
sqe, and a given group ID in sqe->buf_group. When the fd becomes ready,
a free buffer from the specified group is selected. If none are
available, the request is terminated with -ENOBUFS. If successful, the
CQE on completion will contain the buffer ID chosen in the cqe->flags
member, encoded as:
(buffer_id << IORING_CQE_BUFFER_SHIFT) | IORING_CQE_F_BUFFER;
Once a buffer has been consumed by a request, it is no longer available
and must be registered again with IORING_OP_PROVIDE_BUFFERS.
Requests need to support this feature. For now, IORING_OP_READ and
IORING_OP_RECV support it. This is checked on SQE submission, a CQE with
res == -EOPNOTSUPP will be posted if attempted on unsupported requests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-23 23:42:51 +00:00
|
|
|
/* select buffer from sqe->buf_group */
|
|
|
|
|
#define IOSQE_BUFFER_SELECT (1U << IOSQE_BUFFER_SELECT_BIT)
|
io_uring: add option to skip CQE posting
Emitting a CQE is expensive from the kernel perspective. Often, it's
also not convenient for the userspace, spends some cycles on processing
and just complicates the logic. A similar problems goes for linked
requests, where we post an CQE for each request in the link.
Introduce a new flags, IOSQE_CQE_SKIP_SUCCESS, trying to help with it.
When set and a request completed successfully, it won't generate a CQE.
When fails, it produces an CQE, but all following linked requests will
be CQE-less, regardless whether they have IOSQE_CQE_SKIP_SUCCESS or not.
The notion of "fail" is the same as for link failing-cancellation, where
it's opcode dependent, and _usually_ result >= 0 is a success, but not
always.
Linked timeouts are a bit special. When the requests it's linked to was
not attempted to be executed, e.g. failing linked requests, it follows
the description above. Otherwise, whether a linked timeout will post a
completion or not solely depends on IOSQE_CQE_SKIP_SUCCESS of that
linked timeout request. Linked timeout never "fail" during execution, so
for them it's unconditional. It's expected for users to not really care
about the result of it but rely solely on the result of the master
request. Another reason for such a treatment is that it's racy, and the
timeout callback may be running awhile the master request posts its
completion.
use case 1:
If one doesn't care about results of some requests, e.g. normal
timeouts, just set IOSQE_CQE_SKIP_SUCCESS. Error result will still be
posted and need to be handled.
use case 2:
Set IOSQE_CQE_SKIP_SUCCESS for all requests of a link but the last,
and it'll post a completion only for the last one if everything goes
right, otherwise there will be one only one CQE for the first failed
request.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/0220fbe06f7cf99e6fc71b4297bb1cb6c0e89c2c.1636559119.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-11-10 15:49:32 +00:00
|
|
|
/* don't post CQE if request succeeded */
|
|
|
|
|
#define IOSQE_CQE_SKIP_SUCCESS (1U << IOSQE_CQE_SKIP_SUCCESS_BIT)
|
2019-01-11 05:13:58 +00:00
|
|
|
|
2019-01-09 15:59:42 +00:00
|
|
|
/*
|
|
|
|
|
* io_uring_setup() flags
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_IOPOLL (1U << 0) /* io_context is polled */
|
io_uring: add submission polling
This enables an application to do IO, without ever entering the kernel.
By using the SQ ring to fill in new sqes and watching for completions
on the CQ ring, we can submit and reap IOs without doing a single system
call. The kernel side thread will poll for new submissions, and in case
of HIPRI/polled IO, it'll also poll for completions.
By default, we allow 1 second of active spinning. This can by changed
by passing in a different grace period at io_uring_register(2) time.
If the thread exceeds this idle time without having any work to do, it
will set:
sq_ring->flags |= IORING_SQ_NEED_WAKEUP.
The application will have to call io_uring_enter() to start things back
up again. If IO is kept busy, that will never be needed. Basically an
application that has this feature enabled will guard it's
io_uring_enter(2) call with:
read_barrier();
if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP)
io_uring_enter(fd, 0, 0, IORING_ENTER_SQ_WAKEUP);
instead of calling it unconditionally.
It's mandatory to use fixed files with this feature. Failure to do so
will result in the application getting an -EBADF CQ entry when
submitting IO.
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-10 18:22:30 +00:00
|
|
|
#define IORING_SETUP_SQPOLL (1U << 1) /* SQ poll thread */
|
|
|
|
|
#define IORING_SETUP_SQ_AFF (1U << 2) /* sq_thread_cpu is valid */
|
2019-10-04 18:10:03 +00:00
|
|
|
#define IORING_SETUP_CQSIZE (1U << 3) /* app defines CQ size */
|
2019-12-28 22:39:54 +00:00
|
|
|
#define IORING_SETUP_CLAMP (1U << 4) /* clamp SQ/CQ ring sizes */
|
2020-01-28 00:15:48 +00:00
|
|
|
#define IORING_SETUP_ATTACH_WQ (1U << 5) /* attach to existing wq */
|
2020-08-27 14:58:31 +00:00
|
|
|
#define IORING_SETUP_R_DISABLED (1U << 6) /* start with ring disabled */
|
2022-03-10 19:59:35 +00:00
|
|
|
#define IORING_SETUP_SUBMIT_ALL (1U << 7) /* continue submit on error */
|
2022-04-26 01:49:03 +00:00
|
|
|
/*
|
|
|
|
|
* Cooperative task running. When requests complete, they often require
|
|
|
|
|
* forcing the submitter to transition to the kernel to complete. If this
|
|
|
|
|
* flag is set, work will be done when the task transitions anyway, rather
|
|
|
|
|
* than force an inter-processor interrupt reschedule. This avoids interrupting
|
|
|
|
|
* a task running in userspace, and saves an IPI.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_COOP_TASKRUN (1U << 8)
|
2022-04-26 01:49:04 +00:00
|
|
|
/*
|
|
|
|
|
* If COOP_TASKRUN is set, get notified if task work is available for
|
|
|
|
|
* running and a kernel transition would be needed to run it. This sets
|
|
|
|
|
* IORING_SQ_TASKRUN in the sq ring flags. Not valid with COOP_TASKRUN.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_TASKRUN_FLAG (1U << 9)
|
2022-04-01 01:27:52 +00:00
|
|
|
#define IORING_SETUP_SQE128 (1U << 10) /* SQEs are 128 byte */
|
2022-04-26 18:21:23 +00:00
|
|
|
#define IORING_SETUP_CQE32 (1U << 11) /* CQEs are 32 byte */
|
2022-06-16 09:22:08 +00:00
|
|
|
/*
|
|
|
|
|
* Only one task is allowed to submit requests
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_SINGLE_ISSUER (1U << 12)
|
2022-04-01 01:27:52 +00:00
|
|
|
|
2022-08-30 12:50:10 +00:00
|
|
|
/*
|
|
|
|
|
* Defer running task work to get events.
|
|
|
|
|
* Rather than running bits of task work whenever the task transitions
|
|
|
|
|
* try to do it just before it is needed.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_DEFER_TASKRUN (1U << 13)
|
|
|
|
|
|
io_uring: support for user allocated memory for rings/sqes
Currently io_uring applications must call mmap(2) twice to map the rings
themselves, and the sqes array. This works fine, but it does not support
using huge pages to back the rings/sqes.
Provide a way for the application to pass in pre-allocated memory for
the rings/sqes, which can then suitably be allocated from shmfs or
via mmap to get huge page support.
Particularly for larger rings, this reduces the TLBs needed.
If an application wishes to take advantage of that, it must pre-allocate
the memory needed for the sq/cq ring, and the sqes. The former must
be passed in via the io_uring_params->cq_off.user_data field, while the
latter is passed in via the io_uring_params->sq_off.user_data field. Then
it must set IORING_SETUP_NO_MMAP in the io_uring_params->flags field,
and io_uring will then map the existing memory into the kernel for shared
use. The application must not call mmap(2) to map rings as it otherwise
would have, that will now fail with -EINVAL if this setup flag was used.
The pages used for the rings and sqes must be contigious. The intent here
is clearly that huge pages should be used, otherwise the normal setup
procedure works fine as-is. The application may use one huge page for
both the rings and sqes.
Outside of those initialization changes, everything works like it did
before.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-11-05 23:20:54 +00:00
|
|
|
/*
|
|
|
|
|
* Application provides the memory for the rings
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_NO_MMAP (1U << 14)
|
|
|
|
|
|
2023-04-28 16:40:30 +00:00
|
|
|
/*
|
|
|
|
|
* Register the ring fd in itself for use with
|
|
|
|
|
* IORING_REGISTER_USE_REGISTERED_RING; return a registered fd index rather
|
|
|
|
|
* than an fd.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_REGISTERED_FD_ONLY (1U << 15)
|
|
|
|
|
|
2023-08-24 22:53:32 +00:00
|
|
|
/*
|
|
|
|
|
* Removes indirection through the SQ index array.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SETUP_NO_SQARRAY (1U << 16)
|
|
|
|
|
|
2024-11-01 09:19:57 +00:00
|
|
|
/* Use hybrid poll in iopoll process */
|
|
|
|
|
#define IORING_SETUP_HYBRID_IOPOLL (1U << 17)
|
|
|
|
|
|
2022-04-26 08:29:04 +00:00
|
|
|
enum io_uring_op {
|
2019-12-11 22:55:43 +00:00
|
|
|
IORING_OP_NOP,
|
|
|
|
|
IORING_OP_READV,
|
|
|
|
|
IORING_OP_WRITEV,
|
|
|
|
|
IORING_OP_FSYNC,
|
|
|
|
|
IORING_OP_READ_FIXED,
|
|
|
|
|
IORING_OP_WRITE_FIXED,
|
|
|
|
|
IORING_OP_POLL_ADD,
|
|
|
|
|
IORING_OP_POLL_REMOVE,
|
|
|
|
|
IORING_OP_SYNC_FILE_RANGE,
|
|
|
|
|
IORING_OP_SENDMSG,
|
|
|
|
|
IORING_OP_RECVMSG,
|
|
|
|
|
IORING_OP_TIMEOUT,
|
|
|
|
|
IORING_OP_TIMEOUT_REMOVE,
|
|
|
|
|
IORING_OP_ACCEPT,
|
|
|
|
|
IORING_OP_ASYNC_CANCEL,
|
|
|
|
|
IORING_OP_LINK_TIMEOUT,
|
|
|
|
|
IORING_OP_CONNECT,
|
2019-12-10 17:38:56 +00:00
|
|
|
IORING_OP_FALLOCATE,
|
2019-12-11 18:20:36 +00:00
|
|
|
IORING_OP_OPENAT,
|
2019-12-11 21:02:38 +00:00
|
|
|
IORING_OP_CLOSE,
|
2022-09-01 10:54:02 +00:00
|
|
|
IORING_OP_FILES_UPDATE,
|
2019-12-14 04:18:10 +00:00
|
|
|
IORING_OP_STATX,
|
2019-12-22 22:19:35 +00:00
|
|
|
IORING_OP_READ,
|
|
|
|
|
IORING_OP_WRITE,
|
2019-12-26 05:03:45 +00:00
|
|
|
IORING_OP_FADVISE,
|
2019-12-26 05:18:28 +00:00
|
|
|
IORING_OP_MADVISE,
|
2020-01-05 03:19:44 +00:00
|
|
|
IORING_OP_SEND,
|
|
|
|
|
IORING_OP_RECV,
|
2020-01-09 00:59:24 +00:00
|
|
|
IORING_OP_OPENAT2,
|
2020-01-08 22:18:09 +00:00
|
|
|
IORING_OP_EPOLL_CTL,
|
2020-02-24 08:32:45 +00:00
|
|
|
IORING_OP_SPLICE,
|
2020-02-23 23:41:33 +00:00
|
|
|
IORING_OP_PROVIDE_BUFFERS,
|
2020-03-02 23:32:28 +00:00
|
|
|
IORING_OP_REMOVE_BUFFERS,
|
2020-05-17 11:18:06 +00:00
|
|
|
IORING_OP_TEE,
|
2020-09-05 17:14:22 +00:00
|
|
|
IORING_OP_SHUTDOWN,
|
2020-09-28 20:23:58 +00:00
|
|
|
IORING_OP_RENAMEAT,
|
2020-09-28 20:27:37 +00:00
|
|
|
IORING_OP_UNLINKAT,
|
2021-07-08 06:34:45 +00:00
|
|
|
IORING_OP_MKDIRAT,
|
2021-07-08 06:34:46 +00:00
|
|
|
IORING_OP_SYMLINKAT,
|
2021-07-08 06:34:47 +00:00
|
|
|
IORING_OP_LINKAT,
|
2022-03-10 13:27:26 +00:00
|
|
|
IORING_OP_MSG_RING,
|
2022-03-23 15:44:19 +00:00
|
|
|
IORING_OP_FSETXATTR,
|
|
|
|
|
IORING_OP_SETXATTR,
|
2022-03-23 15:44:20 +00:00
|
|
|
IORING_OP_FGETXATTR,
|
|
|
|
|
IORING_OP_GETXATTR,
|
2022-04-12 20:22:40 +00:00
|
|
|
IORING_OP_SOCKET,
|
2022-05-11 05:47:45 +00:00
|
|
|
IORING_OP_URING_CMD,
|
2022-09-01 10:54:04 +00:00
|
|
|
IORING_OP_SEND_ZC,
|
2022-09-21 11:17:54 +00:00
|
|
|
IORING_OP_SENDMSG_ZC,
|
2023-09-11 19:35:42 +00:00
|
|
|
IORING_OP_READ_MULTISHOT,
|
2023-07-10 22:14:37 +00:00
|
|
|
IORING_OP_WAITID,
|
io_uring: add support for futex wake and wait
Add support for FUTEX_WAKE/WAIT primitives.
IORING_OP_FUTEX_WAKE is mix of FUTEX_WAKE and FUTEX_WAKE_BITSET, as
it does support passing in a bitset.
Similary, IORING_OP_FUTEX_WAIT is a mix of FUTEX_WAIT and
FUTEX_WAIT_BITSET.
For both of them, they are using the futex2 interface.
FUTEX_WAKE is straight forward, as those can always be done directly from
the io_uring submission without needing async handling. For FUTEX_WAIT,
things are a bit more complicated. If the futex isn't ready, then we
rely on a callback via futex_queue->wake() when someone wakes up the
futex. From that calback, we queue up task_work with the original task,
which will post a CQE and wake it, if necessary.
Cancelations are supported, both from the application point-of-view,
but also to be able to cancel pending waits if the ring exits before
all events have occurred. The return value of futex_unqueue() is used
to gate who wins the potential race between cancelation and futex
wakeups. Whomever gets a 'ret == 1' return from that claims ownership
of the io_uring futex request.
This is just the barebones wait/wake support. PI or REQUEUE support is
not added at this point, unclear if we might look into that later.
Likewise, explicit timeouts are not supported either. It is expected
that users that need timeouts would do so via the usual io_uring
mechanism to do that using linked timeouts.
The SQE format is as follows:
`addr` Address of futex
`fd` futex2(2) FUTEX2_* flags
`futex_flags` io_uring specific command flags. None valid now.
`addr2` Value of futex
`addr3` Mask to wake/wait
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-06-08 17:57:40 +00:00
|
|
|
IORING_OP_FUTEX_WAIT,
|
|
|
|
|
IORING_OP_FUTEX_WAKE,
|
io_uring: add support for vectored futex waits
This adds support for IORING_OP_FUTEX_WAITV, which allows registering a
notification for a number of futexes at once. If one of the futexes are
woken, then the request will complete with the index of the futex that got
woken as the result. This is identical to what the normal vectored futex
waitv operation does.
Use like IORING_OP_FUTEX_WAIT, except sqe->addr must now contain a
pointer to a struct futex_waitv array, and sqe->off must now contain the
number of elements in that array. As flags are passed in the futex_vector
array, and likewise for the value and futex address(es), sqe->addr2
and sqe->addr3 are also reserved for IORING_OP_FUTEX_WAITV.
For cancelations, FUTEX_WAITV does not rely on the futex_unqueue()
return value as we're dealing with multiple futexes. Instead, a separate
per io_uring request atomic is used to claim ownership of the request.
Waiting on N futexes could be done with IORING_OP_FUTEX_WAIT as well,
but that punts a lot of the work to the application:
1) Application would need to submit N IORING_OP_FUTEX_WAIT requests,
rather than just a single IORING_OP_FUTEX_WAITV.
2) When one futex is woken, application would need to cancel the
remaining N-1 requests that didn't trigger.
While this is of course doable, having a single vectored futex wait
makes for much simpler application code.
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2023-06-13 01:04:32 +00:00
|
|
|
IORING_OP_FUTEX_WAITV,
|
2023-12-08 03:06:02 +00:00
|
|
|
IORING_OP_FIXED_FD_INSTALL,
|
2024-02-02 12:17:24 +00:00
|
|
|
IORING_OP_FTRUNCATE,
|
2024-06-14 16:30:46 +00:00
|
|
|
IORING_OP_BIND,
|
2024-06-14 16:30:47 +00:00
|
|
|
IORING_OP_LISTEN,
|
2025-02-15 00:09:41 +00:00
|
|
|
IORING_OP_RECV_ZC,
|
io_uring/epoll: add support for IORING_OP_EPOLL_WAIT
For existing epoll event loops that can't fully convert to io_uring,
the used approach is usually to add the io_uring fd to the epoll
instance and use epoll_wait() to wait on both "legacy" and io_uring
events. While this work, it isn't optimal as:
1) epoll_wait() is pretty limited in what it can do. It does not support
partial reaping of events, or waiting on a batch of events.
2) When an io_uring ring is added to an epoll instance, it activates the
io_uring "I'm being polled" logic which slows things down.
Rather than use this approach, with EPOLL_WAIT support added to io_uring,
event loops can use the normal io_uring wait logic for everything, as
long as an epoll wait request has been armed with io_uring.
Note that IORING_OP_EPOLL_WAIT does NOT take a timeout value, as this
is an async request. Waiting on io_uring events in general has various
timeout parameters, and those are the ones that should be used when
waiting on any kind of request. If events are immediately available for
reaping, then This opcode will return those immediately. If none are
available, then it will post an async completion when they become
available.
cqe->res will contain either an error code (< 0 value) for a malformed
request, invalid epoll instance, etc. It will return a positive result
indicating how many events were reaped.
IORING_OP_EPOLL_WAIT requests may be canceled using the normal io_uring
cancelation infrastructure.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2025-01-31 21:29:09 +00:00
|
|
|
IORING_OP_EPOLL_WAIT,
|
2025-03-07 16:00:31 +00:00
|
|
|
IORING_OP_READV_FIXED,
|
|
|
|
|
IORING_OP_WRITEV_FIXED,
|
2025-04-04 20:50:59 +00:00
|
|
|
IORING_OP_PIPE,
|
2019-12-11 22:55:43 +00:00
|
|
|
|
|
|
|
|
/* this goes last, obviously */
|
|
|
|
|
IORING_OP_LAST,
|
|
|
|
|
};
|
2019-01-11 16:43:02 +00:00
|
|
|
|
2022-09-30 06:27:39 +00:00
|
|
|
/*
|
2023-09-28 12:43:24 +00:00
|
|
|
* sqe->uring_cmd_flags top 8bits aren't available for userspace
|
2022-11-06 20:17:27 +00:00
|
|
|
* IORING_URING_CMD_FIXED use registered buffer; pass this flag
|
2022-09-30 06:27:39 +00:00
|
|
|
* along with setting sqe->buf_index.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_URING_CMD_FIXED (1U << 0)
|
2023-09-28 12:43:24 +00:00
|
|
|
#define IORING_URING_CMD_MASK IORING_URING_CMD_FIXED
|
2022-09-30 06:27:39 +00:00
|
|
|
|
|
|
|
|
|
2019-01-11 16:43:02 +00:00
|
|
|
/*
|
|
|
|
|
* sqe->fsync_flags
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_FSYNC_DATASYNC (1U << 0)
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
|
2019-10-15 22:48:15 +00:00
|
|
|
/*
|
|
|
|
|
* sqe->timeout_flags
|
|
|
|
|
*/
|
2021-08-29 01:54:39 +00:00
|
|
|
#define IORING_TIMEOUT_ABS (1U << 0)
|
|
|
|
|
#define IORING_TIMEOUT_UPDATE (1U << 1)
|
|
|
|
|
#define IORING_TIMEOUT_BOOTTIME (1U << 2)
|
|
|
|
|
#define IORING_TIMEOUT_REALTIME (1U << 3)
|
|
|
|
|
#define IORING_LINK_TIMEOUT_UPDATE (1U << 4)
|
2021-10-02 18:36:14 +00:00
|
|
|
#define IORING_TIMEOUT_ETIME_SUCCESS (1U << 5)
|
2023-04-18 22:58:18 +00:00
|
|
|
#define IORING_TIMEOUT_MULTISHOT (1U << 6)
|
2021-08-27 23:11:06 +00:00
|
|
|
#define IORING_TIMEOUT_CLOCK_MASK (IORING_TIMEOUT_BOOTTIME | IORING_TIMEOUT_REALTIME)
|
2021-08-29 01:54:39 +00:00
|
|
|
#define IORING_TIMEOUT_UPDATE_MASK (IORING_TIMEOUT_UPDATE | IORING_LINK_TIMEOUT_UPDATE)
|
2020-02-24 08:32:45 +00:00
|
|
|
/*
|
|
|
|
|
* sqe->splice_flags
|
|
|
|
|
* extends splice(2) flags
|
|
|
|
|
*/
|
|
|
|
|
#define SPLICE_F_FD_IN_FIXED (1U << 31) /* the last bit of __u32 */
|
|
|
|
|
|
2021-02-23 05:08:01 +00:00
|
|
|
/*
|
|
|
|
|
* POLL_ADD flags. Note that since sqe->poll_events is the flag space, the
|
|
|
|
|
* command flags for POLL_ADD are stored in sqe->len.
|
|
|
|
|
*
|
|
|
|
|
* IORING_POLL_ADD_MULTI Multishot poll. Sets IORING_CQE_F_MORE if
|
|
|
|
|
* the poll handler will continue to report
|
|
|
|
|
* CQEs on behalf of the same SQE.
|
io_uring: allow events and user_data update of running poll requests
This adds two new POLL_ADD flags, IORING_POLL_UPDATE_EVENTS and
IORING_POLL_UPDATE_USER_DATA. As with the other POLL_ADD flag, these are
masked into sqe->len. If set, the POLL_ADD will have the following
behavior:
- sqe->addr must contain the the user_data of the poll request that
needs to be modified. This field is otherwise invalid for a POLL_ADD
command.
- If IORING_POLL_UPDATE_EVENTS is set, sqe->poll_events must contain the
new mask for the existing poll request. There are no checks for whether
these are identical or not, if a matching poll request is found, then it
is re-armed with the new mask.
- If IORING_POLL_UPDATE_USER_DATA is set, sqe->off must contain the new
user_data for the existing poll request.
A POLL_ADD with any of these flags set may complete with any of the
following results:
1) 0, which means that we successfully found the existing poll request
specified, and performed the re-arm procedure. Any error from that
re-arm will be exposed as a completion event for that original poll
request, not for the update request.
2) -ENOENT, if no existing poll request was found with the given
user_data.
3) -EALREADY, if the existing poll request was already in the process of
being removed/canceled/completing.
4) -EACCES, if an attempt was made to modify an internal poll request
(eg not one originally issued ass IORING_OP_POLL_ADD).
The usual -EINVAL cases apply as well, if any invalid fields are set
in the sqe for this command type.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-03-17 14:37:41 +00:00
|
|
|
*
|
|
|
|
|
* IORING_POLL_UPDATE Update existing poll request, matching
|
|
|
|
|
* sqe->addr as the old user_data field.
|
2022-05-27 16:55:07 +00:00
|
|
|
*
|
|
|
|
|
* IORING_POLL_LEVEL Level triggered poll.
|
2021-02-23 05:08:01 +00:00
|
|
|
*/
|
|
|
|
|
#define IORING_POLL_ADD_MULTI (1U << 0)
|
io_uring: allow events and user_data update of running poll requests
This adds two new POLL_ADD flags, IORING_POLL_UPDATE_EVENTS and
IORING_POLL_UPDATE_USER_DATA. As with the other POLL_ADD flag, these are
masked into sqe->len. If set, the POLL_ADD will have the following
behavior:
- sqe->addr must contain the the user_data of the poll request that
needs to be modified. This field is otherwise invalid for a POLL_ADD
command.
- If IORING_POLL_UPDATE_EVENTS is set, sqe->poll_events must contain the
new mask for the existing poll request. There are no checks for whether
these are identical or not, if a matching poll request is found, then it
is re-armed with the new mask.
- If IORING_POLL_UPDATE_USER_DATA is set, sqe->off must contain the new
user_data for the existing poll request.
A POLL_ADD with any of these flags set may complete with any of the
following results:
1) 0, which means that we successfully found the existing poll request
specified, and performed the re-arm procedure. Any error from that
re-arm will be exposed as a completion event for that original poll
request, not for the update request.
2) -ENOENT, if no existing poll request was found with the given
user_data.
3) -EALREADY, if the existing poll request was already in the process of
being removed/canceled/completing.
4) -EACCES, if an attempt was made to modify an internal poll request
(eg not one originally issued ass IORING_OP_POLL_ADD).
The usual -EINVAL cases apply as well, if any invalid fields are set
in the sqe for this command type.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-03-17 14:37:41 +00:00
|
|
|
#define IORING_POLL_UPDATE_EVENTS (1U << 1)
|
|
|
|
|
#define IORING_POLL_UPDATE_USER_DATA (1U << 2)
|
2022-05-27 16:55:07 +00:00
|
|
|
#define IORING_POLL_ADD_LEVEL (1U << 3)
|
2021-02-23 05:08:01 +00:00
|
|
|
|
2022-04-18 16:44:00 +00:00
|
|
|
/*
|
|
|
|
|
* ASYNC_CANCEL flags.
|
|
|
|
|
*
|
|
|
|
|
* IORING_ASYNC_CANCEL_ALL Cancel all requests that match the given key
|
2022-04-18 16:44:01 +00:00
|
|
|
* IORING_ASYNC_CANCEL_FD Key off 'fd' for cancelation rather than the
|
|
|
|
|
* request 'user_data'
|
2022-04-18 16:44:02 +00:00
|
|
|
* IORING_ASYNC_CANCEL_ANY Match any request
|
2022-06-18 15:47:04 +00:00
|
|
|
* IORING_ASYNC_CANCEL_FD_FIXED 'fd' passed in is a fixed descriptor
|
2023-06-23 16:33:11 +00:00
|
|
|
* IORING_ASYNC_CANCEL_USERDATA Match on user_data, default for no other key
|
2023-06-23 16:36:43 +00:00
|
|
|
* IORING_ASYNC_CANCEL_OP Match request based on opcode
|
2022-04-18 16:44:00 +00:00
|
|
|
*/
|
|
|
|
|
#define IORING_ASYNC_CANCEL_ALL (1U << 0)
|
2022-04-18 16:44:01 +00:00
|
|
|
#define IORING_ASYNC_CANCEL_FD (1U << 1)
|
2022-04-18 16:44:02 +00:00
|
|
|
#define IORING_ASYNC_CANCEL_ANY (1U << 2)
|
2022-06-18 15:47:04 +00:00
|
|
|
#define IORING_ASYNC_CANCEL_FD_FIXED (1U << 3)
|
2023-06-23 16:33:11 +00:00
|
|
|
#define IORING_ASYNC_CANCEL_USERDATA (1U << 4)
|
2023-06-23 16:36:43 +00:00
|
|
|
#define IORING_ASYNC_CANCEL_OP (1U << 5)
|
2022-04-18 16:44:00 +00:00
|
|
|
|
2022-04-26 18:11:33 +00:00
|
|
|
/*
|
2022-06-30 12:25:57 +00:00
|
|
|
* send/sendmsg and recv/recvmsg flags (sqe->ioprio)
|
2022-04-26 18:11:33 +00:00
|
|
|
*
|
|
|
|
|
* IORING_RECVSEND_POLL_FIRST If set, instead of first attempting to send
|
|
|
|
|
* or receive and arm poll if that yields an
|
|
|
|
|
* -EAGAIN result, arm poll upfront and skip
|
|
|
|
|
* the initial transfer attempt.
|
2022-06-30 09:12:29 +00:00
|
|
|
*
|
|
|
|
|
* IORING_RECV_MULTISHOT Multishot recv. Sets IORING_CQE_F_MORE if
|
|
|
|
|
* the handler will continue to report
|
|
|
|
|
* CQEs on behalf of the same SQE.
|
2022-07-12 20:52:46 +00:00
|
|
|
*
|
|
|
|
|
* IORING_RECVSEND_FIXED_BUF Use registered buffers, the index is stored in
|
|
|
|
|
* the buf_index field.
|
2022-10-27 18:34:45 +00:00
|
|
|
*
|
|
|
|
|
* IORING_SEND_ZC_REPORT_USAGE
|
|
|
|
|
* If set, SEND[MSG]_ZC should report
|
|
|
|
|
* the zerocopy usage in cqe.res
|
|
|
|
|
* for the IORING_CQE_F_NOTIF cqe.
|
|
|
|
|
* 0 is reported if zerocopy was actually possible.
|
|
|
|
|
* IORING_NOTIF_USAGE_ZC_COPIED if data was copied
|
|
|
|
|
* (at least partially).
|
2024-03-05 20:10:04 +00:00
|
|
|
*
|
2024-03-05 23:22:04 +00:00
|
|
|
* IORING_RECVSEND_BUNDLE Used with IOSQE_BUFFER_SELECT. If set, send or
|
|
|
|
|
* recv will grab as many buffers from the buffer
|
|
|
|
|
* group ID given and send them all. The completion
|
|
|
|
|
* result will be the number of buffers send, with
|
|
|
|
|
* the starting buffer ID in cqe->flags as per
|
|
|
|
|
* usual for provided buffer usage. The buffers
|
2025-02-18 23:47:40 +00:00
|
|
|
* will be contiguous from the starting buffer ID.
|
2022-04-26 18:11:33 +00:00
|
|
|
*/
|
|
|
|
|
#define IORING_RECVSEND_POLL_FIRST (1U << 0)
|
2022-07-12 20:52:46 +00:00
|
|
|
#define IORING_RECV_MULTISHOT (1U << 1)
|
|
|
|
|
#define IORING_RECVSEND_FIXED_BUF (1U << 2)
|
2022-10-27 18:34:45 +00:00
|
|
|
#define IORING_SEND_ZC_REPORT_USAGE (1U << 3)
|
2024-03-05 20:10:04 +00:00
|
|
|
#define IORING_RECVSEND_BUNDLE (1U << 4)
|
2022-10-27 18:34:45 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* cqe.res for IORING_CQE_F_NOTIF if
|
|
|
|
|
* IORING_SEND_ZC_REPORT_USAGE was requested
|
|
|
|
|
*
|
|
|
|
|
* It should be treated as a flag, all other
|
|
|
|
|
* bits of cqe.res should be treated as reserved!
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_NOTIF_USAGE_ZC_COPIED (1U << 31)
|
2022-04-26 18:11:33 +00:00
|
|
|
|
2022-05-14 14:20:43 +00:00
|
|
|
/*
|
|
|
|
|
* accept flags stored in sqe->ioprio
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_ACCEPT_MULTISHOT (1U << 0)
|
2024-05-07 20:06:15 +00:00
|
|
|
#define IORING_ACCEPT_DONTWAIT (1U << 1)
|
2024-05-08 14:17:50 +00:00
|
|
|
#define IORING_ACCEPT_POLL_FIRST (1U << 2)
|
2022-05-14 14:20:43 +00:00
|
|
|
|
io_uring: add support for passing fixed file descriptors
With IORING_OP_MSG_RING, one ring can send a message to another ring.
Extend that support to also allow sending a fixed file descriptor to
that ring, enabling one ring to pass a registered descriptor to another
one.
Arguments are extended to pass in:
sqe->addr3 fixed file slot in source ring
sqe->file_index fixed file slot in destination ring
IORING_OP_MSG_RING is extended to take a command argument in sqe->addr.
If set to zero (or IORING_MSG_DATA), it sends just a message like before.
If set to IORING_MSG_SEND_FD, a fixed file descriptor is sent according
to the above arguments.
Two common use cases for this are:
1) Server needs to be shutdown or restarted, pass file descriptors to
another onei
2) Backend is split, and one accepts connections, while others then get
the fd passed and handle the actual connection.
Both of those are classic SCM_RIGHTS use cases, and it's not possible to
support them with direct descriptors today.
By default, this will post a CQE to the target ring, similarly to how
IORING_MSG_DATA does it. If IORING_MSG_RING_CQE_SKIP is set, no message
is posted to the target ring. The issuer is expected to notify the
receiver side separately.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-13 10:47:02 +00:00
|
|
|
/*
|
|
|
|
|
* IORING_OP_MSG_RING command types, stored in sqe->addr
|
|
|
|
|
*/
|
2024-03-28 21:09:35 +00:00
|
|
|
enum io_uring_msg_ring_flags {
|
io_uring: add support for passing fixed file descriptors
With IORING_OP_MSG_RING, one ring can send a message to another ring.
Extend that support to also allow sending a fixed file descriptor to
that ring, enabling one ring to pass a registered descriptor to another
one.
Arguments are extended to pass in:
sqe->addr3 fixed file slot in source ring
sqe->file_index fixed file slot in destination ring
IORING_OP_MSG_RING is extended to take a command argument in sqe->addr.
If set to zero (or IORING_MSG_DATA), it sends just a message like before.
If set to IORING_MSG_SEND_FD, a fixed file descriptor is sent according
to the above arguments.
Two common use cases for this are:
1) Server needs to be shutdown or restarted, pass file descriptors to
another onei
2) Backend is split, and one accepts connections, while others then get
the fd passed and handle the actual connection.
Both of those are classic SCM_RIGHTS use cases, and it's not possible to
support them with direct descriptors today.
By default, this will post a CQE to the target ring, similarly to how
IORING_MSG_DATA does it. If IORING_MSG_RING_CQE_SKIP is set, no message
is posted to the target ring. The issuer is expected to notify the
receiver side separately.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-13 10:47:02 +00:00
|
|
|
IORING_MSG_DATA, /* pass sqe->len as 'res' and off as user_data */
|
|
|
|
|
IORING_MSG_SEND_FD, /* send a registered fd to another ring */
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* IORING_OP_MSG_RING flags (sqe->msg_ring_flags)
|
|
|
|
|
*
|
|
|
|
|
* IORING_MSG_RING_CQE_SKIP Don't post a CQE to the target ring. Not
|
|
|
|
|
* applicable for IORING_MSG_DATA, obviously.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_MSG_RING_CQE_SKIP (1U << 0)
|
2023-01-03 16:05:07 +00:00
|
|
|
/* Pass through the flags from sqe->file_index to cqe->flags */
|
|
|
|
|
#define IORING_MSG_RING_FLAGS_PASS (1U << 1)
|
io_uring: add support for passing fixed file descriptors
With IORING_OP_MSG_RING, one ring can send a message to another ring.
Extend that support to also allow sending a fixed file descriptor to
that ring, enabling one ring to pass a registered descriptor to another
one.
Arguments are extended to pass in:
sqe->addr3 fixed file slot in source ring
sqe->file_index fixed file slot in destination ring
IORING_OP_MSG_RING is extended to take a command argument in sqe->addr.
If set to zero (or IORING_MSG_DATA), it sends just a message like before.
If set to IORING_MSG_SEND_FD, a fixed file descriptor is sent according
to the above arguments.
Two common use cases for this are:
1) Server needs to be shutdown or restarted, pass file descriptors to
another onei
2) Backend is split, and one accepts connections, while others then get
the fd passed and handle the actual connection.
Both of those are classic SCM_RIGHTS use cases, and it's not possible to
support them with direct descriptors today.
By default, this will post a CQE to the target ring, similarly to how
IORING_MSG_DATA does it. If IORING_MSG_RING_CQE_SKIP is set, no message
is posted to the target ring. The issuer is expected to notify the
receiver side separately.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-13 10:47:02 +00:00
|
|
|
|
2023-12-08 03:06:02 +00:00
|
|
|
/*
|
|
|
|
|
* IORING_OP_FIXED_FD_INSTALL flags (sqe->install_fd_flags)
|
|
|
|
|
*
|
|
|
|
|
* IORING_FIXED_FD_NO_CLOEXEC Don't mark the fd as O_CLOEXEC
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_FIXED_FD_NO_CLOEXEC (1U << 0)
|
|
|
|
|
|
2024-05-10 03:50:28 +00:00
|
|
|
/*
|
|
|
|
|
* IORING_OP_NOP flags (sqe->nop_flags)
|
|
|
|
|
*
|
|
|
|
|
* IORING_NOP_INJECT_RESULT Inject result from sqe->result
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_NOP_INJECT_RESULT (1U << 0)
|
2024-10-27 14:59:10 +00:00
|
|
|
#define IORING_NOP_FILE (1U << 1)
|
|
|
|
|
#define IORING_NOP_FIXED_FILE (1U << 2)
|
|
|
|
|
#define IORING_NOP_FIXED_BUFFER (1U << 3)
|
2024-05-10 03:50:28 +00:00
|
|
|
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
/*
|
|
|
|
|
* IO completion data structure (Completion Queue Entry)
|
|
|
|
|
*/
|
|
|
|
|
struct io_uring_cqe {
|
2024-08-16 18:15:23 +00:00
|
|
|
__u64 user_data; /* sqe->user_data value passed back */
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
__s32 res; /* result code for this event */
|
|
|
|
|
__u32 flags;
|
2022-04-26 18:21:23 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If the ring is initialized with IORING_SETUP_CQE32, then this field
|
|
|
|
|
* contains 16-bytes of padding, doubling the size of the CQE.
|
|
|
|
|
*/
|
|
|
|
|
__u64 big_cqe[];
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
};
|
|
|
|
|
|
io_uring: support buffer selection for OP_READ and OP_RECV
If a server process has tons of pending socket connections, generally
it uses epoll to wait for activity. When the socket is ready for reading
(or writing), the task can select a buffer and issue a recv/send on the
given fd.
Now that we have fast (non-async thread) support, a task can have tons
of pending reads or writes pending. But that means they need buffers to
back that data, and if the number of connections is high enough, having
them preallocated for all possible connections is unfeasible.
With IORING_OP_PROVIDE_BUFFERS, an application can register buffers to
use for any request. The request then sets IOSQE_BUFFER_SELECT in the
sqe, and a given group ID in sqe->buf_group. When the fd becomes ready,
a free buffer from the specified group is selected. If none are
available, the request is terminated with -ENOBUFS. If successful, the
CQE on completion will contain the buffer ID chosen in the cqe->flags
member, encoded as:
(buffer_id << IORING_CQE_BUFFER_SHIFT) | IORING_CQE_F_BUFFER;
Once a buffer has been consumed by a request, it is no longer available
and must be registered again with IORING_OP_PROVIDE_BUFFERS.
Requests need to support this feature. For now, IORING_OP_READ and
IORING_OP_RECV support it. This is checked on SQE submission, a CQE with
res == -EOPNOTSUPP will be posted if attempted on unsupported requests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-23 23:42:51 +00:00
|
|
|
/*
|
|
|
|
|
* cqe->flags
|
|
|
|
|
*
|
|
|
|
|
* IORING_CQE_F_BUFFER If set, the upper 16 bits are the buffer ID
|
2021-02-23 05:08:01 +00:00
|
|
|
* IORING_CQE_F_MORE If set, parent SQE will generate more CQE entries
|
2022-04-26 23:39:50 +00:00
|
|
|
* IORING_CQE_F_SOCK_NONEMPTY If set, more data to read after socket recv
|
2022-09-01 10:54:04 +00:00
|
|
|
* IORING_CQE_F_NOTIF Set for notification CQEs. Can be used to distinct
|
|
|
|
|
* them from sends.
|
io_uring/kbuf: add support for incremental buffer consumption
By default, any recv/read operation that uses provided buffers will
consume at least 1 buffer fully (and maybe more, in case of bundles).
This adds support for incremental consumption, meaning that an
application may add large buffers, and each read/recv will just consume
the part of the buffer that it needs.
For example, let's say an application registers 1MB buffers in a
provided buffer ring, for streaming receives. If it gets a short recv,
then the full 1MB buffer will be consumed and passed back to the
application. With incremental consumption, only the part that was
actually used is consumed, and the buffer remains the current one.
This means that both the application and the kernel needs to keep track
of what the current receive point is. Each recv will still pass back a
buffer ID and the size consumed, the only difference is that before the
next receive would always be the next buffer in the ring. Now the same
buffer ID may return multiple receives, each at an offset into that
buffer from where the previous receive left off. Example:
Application registers a provided buffer ring, and adds two 32K buffers
to the ring.
Buffer1 address: 0x1000000 (buffer ID 0)
Buffer2 address: 0x2000000 (buffer ID 1)
A recv completion is received with the following values:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
and the application now knows that 4096b of data is available at
0x1000000, the start of that buffer, and that more data from this buffer
will be coming. Now the next receive comes in:
cqe->res 0x2010 (8k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
which tells the application that 8k is available where the last
completion left off, at 0x1001000. Next completion is:
cqe->res 0x5000 (20k bytes received)
cqe->flags 0x1 (CQE_F_BUFFER set, buffer ID 0)
and the application now knows that 20k of data is available at
0x1003000, which is where the previous receive ended. CQE_F_BUF_MORE
isn't set, as no more data is available in this buffer ID. The next
completion is then:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x10001 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 1)
which tells the application that buffer ID 1 is now the current one,
hence there's 4k of valid data at 0x2000000. 0x2001000 will be the next
receive point for this buffer ID.
When a buffer will be reused by future CQE completions,
IORING_CQE_BUF_MORE will be set in cqe->flags. This tells the application
that the kernel isn't done with the buffer yet, and that it should expect
more completions for this buffer ID. Will only be set by provided buffer
rings setup with IOU_PBUF_RING INC, as that's the only type of buffer
that will see multiple consecutive completions for the same buffer ID.
For any other provided buffer type, any completion that passes back
a buffer to the application is final.
Once a buffer has been fully consumed, the buffer ring head is
incremented and the next receive will indicate the next buffer ID in the
CQE cflags.
On the send side, the application can manage how much data is sent from
an existing buffer by setting sqe->len to the desired send length.
An application can request incremental consumption by setting
IOU_PBUF_RING_INC in the provided buffer ring registration. Outside of
that, any provided buffer ring setup and buffer additions is done like
before, no changes there. The only change is in how an application may
see multiple completions for the same buffer ID, hence needing to know
where the next receive will happen.
Note that like existing provided buffer rings, this should not be used
with IOSQE_ASYNC, as both really require the ring to remain locked over
the duration of the buffer selection and the operation completion. It
will consume a buffer otherwise regardless of the size of the IO done.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-08-09 17:20:45 +00:00
|
|
|
* IORING_CQE_F_BUF_MORE If set, the buffer ID set in the completion will get
|
|
|
|
|
* more completions. In other words, the buffer is being
|
|
|
|
|
* partially consumed, and will be used by the kernel for
|
|
|
|
|
* more completions. This is only set for buffers used via
|
|
|
|
|
* the incremental buffer consumption, as provided by
|
|
|
|
|
* a ring buffer setup with IOU_PBUF_RING_INC. For any
|
|
|
|
|
* other provided buffer type, all completions with a
|
|
|
|
|
* buffer passed back is automatically returned to the
|
|
|
|
|
* application.
|
io_uring: support buffer selection for OP_READ and OP_RECV
If a server process has tons of pending socket connections, generally
it uses epoll to wait for activity. When the socket is ready for reading
(or writing), the task can select a buffer and issue a recv/send on the
given fd.
Now that we have fast (non-async thread) support, a task can have tons
of pending reads or writes pending. But that means they need buffers to
back that data, and if the number of connections is high enough, having
them preallocated for all possible connections is unfeasible.
With IORING_OP_PROVIDE_BUFFERS, an application can register buffers to
use for any request. The request then sets IOSQE_BUFFER_SELECT in the
sqe, and a given group ID in sqe->buf_group. When the fd becomes ready,
a free buffer from the specified group is selected. If none are
available, the request is terminated with -ENOBUFS. If successful, the
CQE on completion will contain the buffer ID chosen in the cqe->flags
member, encoded as:
(buffer_id << IORING_CQE_BUFFER_SHIFT) | IORING_CQE_F_BUFFER;
Once a buffer has been consumed by a request, it is no longer available
and must be registered again with IORING_OP_PROVIDE_BUFFERS.
Requests need to support this feature. For now, IORING_OP_READ and
IORING_OP_RECV support it. This is checked on SQE submission, a CQE with
res == -EOPNOTSUPP will be posted if attempted on unsupported requests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-23 23:42:51 +00:00
|
|
|
*/
|
|
|
|
|
#define IORING_CQE_F_BUFFER (1U << 0)
|
2021-02-23 05:08:01 +00:00
|
|
|
#define IORING_CQE_F_MORE (1U << 1)
|
2022-04-26 23:39:50 +00:00
|
|
|
#define IORING_CQE_F_SOCK_NONEMPTY (1U << 2)
|
2022-09-01 10:54:04 +00:00
|
|
|
#define IORING_CQE_F_NOTIF (1U << 3)
|
io_uring/kbuf: add support for incremental buffer consumption
By default, any recv/read operation that uses provided buffers will
consume at least 1 buffer fully (and maybe more, in case of bundles).
This adds support for incremental consumption, meaning that an
application may add large buffers, and each read/recv will just consume
the part of the buffer that it needs.
For example, let's say an application registers 1MB buffers in a
provided buffer ring, for streaming receives. If it gets a short recv,
then the full 1MB buffer will be consumed and passed back to the
application. With incremental consumption, only the part that was
actually used is consumed, and the buffer remains the current one.
This means that both the application and the kernel needs to keep track
of what the current receive point is. Each recv will still pass back a
buffer ID and the size consumed, the only difference is that before the
next receive would always be the next buffer in the ring. Now the same
buffer ID may return multiple receives, each at an offset into that
buffer from where the previous receive left off. Example:
Application registers a provided buffer ring, and adds two 32K buffers
to the ring.
Buffer1 address: 0x1000000 (buffer ID 0)
Buffer2 address: 0x2000000 (buffer ID 1)
A recv completion is received with the following values:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
and the application now knows that 4096b of data is available at
0x1000000, the start of that buffer, and that more data from this buffer
will be coming. Now the next receive comes in:
cqe->res 0x2010 (8k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
which tells the application that 8k is available where the last
completion left off, at 0x1001000. Next completion is:
cqe->res 0x5000 (20k bytes received)
cqe->flags 0x1 (CQE_F_BUFFER set, buffer ID 0)
and the application now knows that 20k of data is available at
0x1003000, which is where the previous receive ended. CQE_F_BUF_MORE
isn't set, as no more data is available in this buffer ID. The next
completion is then:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x10001 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 1)
which tells the application that buffer ID 1 is now the current one,
hence there's 4k of valid data at 0x2000000. 0x2001000 will be the next
receive point for this buffer ID.
When a buffer will be reused by future CQE completions,
IORING_CQE_BUF_MORE will be set in cqe->flags. This tells the application
that the kernel isn't done with the buffer yet, and that it should expect
more completions for this buffer ID. Will only be set by provided buffer
rings setup with IOU_PBUF_RING INC, as that's the only type of buffer
that will see multiple consecutive completions for the same buffer ID.
For any other provided buffer type, any completion that passes back
a buffer to the application is final.
Once a buffer has been fully consumed, the buffer ring head is
incremented and the next receive will indicate the next buffer ID in the
CQE cflags.
On the send side, the application can manage how much data is sent from
an existing buffer by setting sqe->len to the desired send length.
An application can request incremental consumption by setting
IOU_PBUF_RING_INC in the provided buffer ring registration. Outside of
that, any provided buffer ring setup and buffer additions is done like
before, no changes there. The only change is in how an application may
see multiple completions for the same buffer ID, hence needing to know
where the next receive will happen.
Note that like existing provided buffer rings, this should not be used
with IOSQE_ASYNC, as both really require the ring to remain locked over
the duration of the buffer selection and the operation completion. It
will consume a buffer otherwise regardless of the size of the IO done.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-08-09 17:20:45 +00:00
|
|
|
#define IORING_CQE_F_BUF_MORE (1U << 4)
|
io_uring: support buffer selection for OP_READ and OP_RECV
If a server process has tons of pending socket connections, generally
it uses epoll to wait for activity. When the socket is ready for reading
(or writing), the task can select a buffer and issue a recv/send on the
given fd.
Now that we have fast (non-async thread) support, a task can have tons
of pending reads or writes pending. But that means they need buffers to
back that data, and if the number of connections is high enough, having
them preallocated for all possible connections is unfeasible.
With IORING_OP_PROVIDE_BUFFERS, an application can register buffers to
use for any request. The request then sets IOSQE_BUFFER_SELECT in the
sqe, and a given group ID in sqe->buf_group. When the fd becomes ready,
a free buffer from the specified group is selected. If none are
available, the request is terminated with -ENOBUFS. If successful, the
CQE on completion will contain the buffer ID chosen in the cqe->flags
member, encoded as:
(buffer_id << IORING_CQE_BUFFER_SHIFT) | IORING_CQE_F_BUFFER;
Once a buffer has been consumed by a request, it is no longer available
and must be registered again with IORING_OP_PROVIDE_BUFFERS.
Requests need to support this feature. For now, IORING_OP_READ and
IORING_OP_RECV support it. This is checked on SQE submission, a CQE with
res == -EOPNOTSUPP will be posted if attempted on unsupported requests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-23 23:42:51 +00:00
|
|
|
|
2024-03-28 21:09:35 +00:00
|
|
|
#define IORING_CQE_BUFFER_SHIFT 16
|
io_uring: support buffer selection for OP_READ and OP_RECV
If a server process has tons of pending socket connections, generally
it uses epoll to wait for activity. When the socket is ready for reading
(or writing), the task can select a buffer and issue a recv/send on the
given fd.
Now that we have fast (non-async thread) support, a task can have tons
of pending reads or writes pending. But that means they need buffers to
back that data, and if the number of connections is high enough, having
them preallocated for all possible connections is unfeasible.
With IORING_OP_PROVIDE_BUFFERS, an application can register buffers to
use for any request. The request then sets IOSQE_BUFFER_SELECT in the
sqe, and a given group ID in sqe->buf_group. When the fd becomes ready,
a free buffer from the specified group is selected. If none are
available, the request is terminated with -ENOBUFS. If successful, the
CQE on completion will contain the buffer ID chosen in the cqe->flags
member, encoded as:
(buffer_id << IORING_CQE_BUFFER_SHIFT) | IORING_CQE_F_BUFFER;
Once a buffer has been consumed by a request, it is no longer available
and must be registered again with IORING_OP_PROVIDE_BUFFERS.
Requests need to support this feature. For now, IORING_OP_READ and
IORING_OP_RECV support it. This is checked on SQE submission, a CQE with
res == -EOPNOTSUPP will be posted if attempted on unsupported requests.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-23 23:42:51 +00:00
|
|
|
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
/*
|
|
|
|
|
* Magic offsets for the application to mmap the data it needs
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_OFF_SQ_RING 0ULL
|
|
|
|
|
#define IORING_OFF_CQ_RING 0x8000000ULL
|
|
|
|
|
#define IORING_OFF_SQES 0x10000000ULL
|
2023-03-14 17:07:19 +00:00
|
|
|
#define IORING_OFF_PBUF_RING 0x80000000ULL
|
|
|
|
|
#define IORING_OFF_PBUF_SHIFT 16
|
|
|
|
|
#define IORING_OFF_MMAP_MASK 0xf8000000ULL
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Filled with the offset for mmap(2)
|
|
|
|
|
*/
|
|
|
|
|
struct io_sqring_offsets {
|
|
|
|
|
__u32 head;
|
|
|
|
|
__u32 tail;
|
|
|
|
|
__u32 ring_mask;
|
|
|
|
|
__u32 ring_entries;
|
|
|
|
|
__u32 flags;
|
|
|
|
|
__u32 dropped;
|
|
|
|
|
__u32 array;
|
|
|
|
|
__u32 resv1;
|
io_uring: support for user allocated memory for rings/sqes
Currently io_uring applications must call mmap(2) twice to map the rings
themselves, and the sqes array. This works fine, but it does not support
using huge pages to back the rings/sqes.
Provide a way for the application to pass in pre-allocated memory for
the rings/sqes, which can then suitably be allocated from shmfs or
via mmap to get huge page support.
Particularly for larger rings, this reduces the TLBs needed.
If an application wishes to take advantage of that, it must pre-allocate
the memory needed for the sq/cq ring, and the sqes. The former must
be passed in via the io_uring_params->cq_off.user_data field, while the
latter is passed in via the io_uring_params->sq_off.user_data field. Then
it must set IORING_SETUP_NO_MMAP in the io_uring_params->flags field,
and io_uring will then map the existing memory into the kernel for shared
use. The application must not call mmap(2) to map rings as it otherwise
would have, that will now fail with -EINVAL if this setup flag was used.
The pages used for the rings and sqes must be contigious. The intent here
is clearly that huge pages should be used, otherwise the normal setup
procedure works fine as-is. The application may use one huge page for
both the rings and sqes.
Outside of those initialization changes, everything works like it did
before.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-11-05 23:20:54 +00:00
|
|
|
__u64 user_addr;
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
};
|
|
|
|
|
|
io_uring: add submission polling
This enables an application to do IO, without ever entering the kernel.
By using the SQ ring to fill in new sqes and watching for completions
on the CQ ring, we can submit and reap IOs without doing a single system
call. The kernel side thread will poll for new submissions, and in case
of HIPRI/polled IO, it'll also poll for completions.
By default, we allow 1 second of active spinning. This can by changed
by passing in a different grace period at io_uring_register(2) time.
If the thread exceeds this idle time without having any work to do, it
will set:
sq_ring->flags |= IORING_SQ_NEED_WAKEUP.
The application will have to call io_uring_enter() to start things back
up again. If IO is kept busy, that will never be needed. Basically an
application that has this feature enabled will guard it's
io_uring_enter(2) call with:
read_barrier();
if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP)
io_uring_enter(fd, 0, 0, IORING_ENTER_SQ_WAKEUP);
instead of calling it unconditionally.
It's mandatory to use fixed files with this feature. Failure to do so
will result in the application getting an -EBADF CQ entry when
submitting IO.
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-10 18:22:30 +00:00
|
|
|
/*
|
|
|
|
|
* sq_ring->flags
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_SQ_NEED_WAKEUP (1U << 0) /* needs io_uring_enter wakeup */
|
io_uring: export cq overflow status to userspace
For those applications which are not willing to use io_uring_enter()
to reap and handle cqes, they may completely rely on liburing's
io_uring_peek_cqe(), but if cq ring has overflowed, currently because
io_uring_peek_cqe() is not aware of this overflow, it won't enter
kernel to flush cqes, below test program can reveal this bug:
static void test_cq_overflow(struct io_uring *ring)
{
struct io_uring_cqe *cqe;
struct io_uring_sqe *sqe;
int issued = 0;
int ret = 0;
do {
sqe = io_uring_get_sqe(ring);
if (!sqe) {
fprintf(stderr, "get sqe failed\n");
break;;
}
ret = io_uring_submit(ring);
if (ret <= 0) {
if (ret != -EBUSY)
fprintf(stderr, "sqe submit failed: %d\n", ret);
break;
}
issued++;
} while (ret > 0);
assert(ret == -EBUSY);
printf("issued requests: %d\n", issued);
while (issued) {
ret = io_uring_peek_cqe(ring, &cqe);
if (ret) {
if (ret != -EAGAIN) {
fprintf(stderr, "peek completion failed: %s\n",
strerror(ret));
break;
}
printf("left requets: %d\n", issued);
continue;
}
io_uring_cqe_seen(ring, cqe);
issued--;
printf("left requets: %d\n", issued);
}
}
int main(int argc, char *argv[])
{
int ret;
struct io_uring ring;
ret = io_uring_queue_init(16, &ring, 0);
if (ret) {
fprintf(stderr, "ring setup failed: %d\n", ret);
return 1;
}
test_cq_overflow(&ring);
return 0;
}
To fix this issue, export cq overflow status to userspace by adding new
IORING_SQ_CQ_OVERFLOW flag, then helper functions() in liburing, such as
io_uring_peek_cqe, can be aware of this cq overflow and do flush accordingly.
Signed-off-by: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-07-09 01:15:29 +00:00
|
|
|
#define IORING_SQ_CQ_OVERFLOW (1U << 1) /* CQ ring is overflown */
|
2022-04-26 01:49:04 +00:00
|
|
|
#define IORING_SQ_TASKRUN (1U << 2) /* task should enter the kernel */
|
io_uring: add submission polling
This enables an application to do IO, without ever entering the kernel.
By using the SQ ring to fill in new sqes and watching for completions
on the CQ ring, we can submit and reap IOs without doing a single system
call. The kernel side thread will poll for new submissions, and in case
of HIPRI/polled IO, it'll also poll for completions.
By default, we allow 1 second of active spinning. This can by changed
by passing in a different grace period at io_uring_register(2) time.
If the thread exceeds this idle time without having any work to do, it
will set:
sq_ring->flags |= IORING_SQ_NEED_WAKEUP.
The application will have to call io_uring_enter() to start things back
up again. If IO is kept busy, that will never be needed. Basically an
application that has this feature enabled will guard it's
io_uring_enter(2) call with:
read_barrier();
if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP)
io_uring_enter(fd, 0, 0, IORING_ENTER_SQ_WAKEUP);
instead of calling it unconditionally.
It's mandatory to use fixed files with this feature. Failure to do so
will result in the application getting an -EBADF CQ entry when
submitting IO.
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-10 18:22:30 +00:00
|
|
|
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
struct io_cqring_offsets {
|
|
|
|
|
__u32 head;
|
|
|
|
|
__u32 tail;
|
|
|
|
|
__u32 ring_mask;
|
|
|
|
|
__u32 ring_entries;
|
|
|
|
|
__u32 overflow;
|
|
|
|
|
__u32 cqes;
|
2020-05-15 16:38:04 +00:00
|
|
|
__u32 flags;
|
|
|
|
|
__u32 resv1;
|
io_uring: support for user allocated memory for rings/sqes
Currently io_uring applications must call mmap(2) twice to map the rings
themselves, and the sqes array. This works fine, but it does not support
using huge pages to back the rings/sqes.
Provide a way for the application to pass in pre-allocated memory for
the rings/sqes, which can then suitably be allocated from shmfs or
via mmap to get huge page support.
Particularly for larger rings, this reduces the TLBs needed.
If an application wishes to take advantage of that, it must pre-allocate
the memory needed for the sq/cq ring, and the sqes. The former must
be passed in via the io_uring_params->cq_off.user_data field, while the
latter is passed in via the io_uring_params->sq_off.user_data field. Then
it must set IORING_SETUP_NO_MMAP in the io_uring_params->flags field,
and io_uring will then map the existing memory into the kernel for shared
use. The application must not call mmap(2) to map rings as it otherwise
would have, that will now fail with -EINVAL if this setup flag was used.
The pages used for the rings and sqes must be contigious. The intent here
is clearly that huge pages should be used, otherwise the normal setup
procedure works fine as-is. The application may use one huge page for
both the rings and sqes.
Outside of those initialization changes, everything works like it did
before.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-11-05 23:20:54 +00:00
|
|
|
__u64 user_addr;
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
};
|
|
|
|
|
|
2020-05-15 16:38:05 +00:00
|
|
|
/*
|
|
|
|
|
* cq_ring->flags
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/* disable eventfd notifications */
|
|
|
|
|
#define IORING_CQ_EVENTFD_DISABLED (1U << 0)
|
|
|
|
|
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
/*
|
|
|
|
|
* io_uring_enter(2) flags
|
|
|
|
|
*/
|
io_uring: add support for registering ring file descriptors
Lots of workloads use multiple threads, in which case the file table is
shared between them. This makes getting and putting the ring file
descriptor for each io_uring_enter(2) system call more expensive, as it
involves an atomic get and put for each call.
Similarly to how we allow registering normal file descriptors to avoid
this overhead, add support for an io_uring_register(2) API that allows
to register the ring fds themselves:
1) IORING_REGISTER_RING_FDS - takes an array of io_uring_rsrc_update
structs, and registers them with the task.
2) IORING_UNREGISTER_RING_FDS - takes an array of io_uring_src_update
structs, and unregisters them.
When a ring fd is registered, it is internally represented by an offset.
This offset is returned to the application, and the application then
uses this offset and sets IORING_ENTER_REGISTERED_RING for the
io_uring_enter(2) system call. This works just like using a registered
file descriptor, rather than a real one, in an SQE, where
IOSQE_FIXED_FILE gets set to tell io_uring that we're using an internal
offset/descriptor rather than a real file descriptor.
In initial testing, this provides a nice bump in performance for
threaded applications in real world cases where the batch count (eg
number of requests submitted per io_uring_enter(2) invocation) is low.
In a microbenchmark, submitting NOP requests, we see the following
increases in performance:
Requests per syscall Baseline Registered Increase
----------------------------------------------------------------
1 ~7030K ~8080K +15%
2 ~13120K ~14800K +13%
4 ~22740K ~25300K +11%
Co-developed-by: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-03-04 15:22:22 +00:00
|
|
|
#define IORING_ENTER_GETEVENTS (1U << 0)
|
|
|
|
|
#define IORING_ENTER_SQ_WAKEUP (1U << 1)
|
|
|
|
|
#define IORING_ENTER_SQ_WAIT (1U << 2)
|
|
|
|
|
#define IORING_ENTER_EXT_ARG (1U << 3)
|
|
|
|
|
#define IORING_ENTER_REGISTERED_RING (1U << 4)
|
2024-08-07 14:18:13 +00:00
|
|
|
#define IORING_ENTER_ABS_TIMER (1U << 5)
|
io_uring: add support for fixed wait regions
Generally applications have 1 or a few waits of waiting, yet they pass
in a struct io_uring_getevents_arg every time. This needs to get copied
and, in turn, the timeout value needs to get copied.
Rather than do this for every invocation, allow the application to
register a fixed set of wait regions that can simply be indexed when
asking the kernel to wait on events.
At ring setup time, the application can register a number of these wait
regions and initialize region/index 0 upfront:
struct io_uring_reg_wait *reg;
reg = io_uring_setup_reg_wait(ring, nr_regions, &ret);
/* set timeout and mark as set, sigmask/sigmask_sz as needed */
reg->ts.tv_sec = 0;
reg->ts.tv_nsec = 100000;
reg->flags = IORING_REG_WAIT_TS;
where nr_regions >= 1 && nr_regions <= PAGE_SIZE / sizeof(*reg). The
above initializes index 0, but 63 other regions can be initialized,
if needed. Now, instead of doing:
struct __kernel_timespec timeout = { .tv_nsec = 100000, };
io_uring_submit_and_wait_timeout(ring, &cqe, nr, &t, NULL);
to wait for events for each submit_and_wait, or just wait, operation, it
can just reference the above region at offset 0 and do:
io_uring_submit_and_wait_reg(ring, &cqe, nr, 0);
to achieve the same goal of waiting 100usec without needing to copy
both struct io_uring_getevents_arg (24b) and struct __kernel_timeout
(16b) for each invocation. Struct io_uring_reg_wait looks as follows:
struct io_uring_reg_wait {
struct __kernel_timespec ts;
__u32 min_wait_usec;
__u32 flags;
__u64 sigmask;
__u32 sigmask_sz;
__u32 pad[3];
__u64 pad2[2];
};
embedding the timeout itself in the region, rather than passing it as
a pointer as well. Note that the signal mask is still passed as a
pointer, both for compatability reasons, but also because there doesn't
seem to be a lot of high frequency waits scenarios that involve setting
and resetting the signal mask for each wait.
The application is free to modify any region before a wait call, or it
can use keep multiple regions with different settings to avoid needing to
modify the same one for wait calls. Up to a page size of regions is mapped
by default, allowing PAGE_SIZE / 64 available regions for use.
The registered region must fit within a page. On a 4kb page size system,
that allows for 64 wait regions if a full page is used, as the size of
struct io_uring_reg_wait is 64b. The region registered must be aligned
to io_uring_reg_wait in size. It's valid to register less than 64
entries.
In network performance testing with zero-copy, this reduced the time
spent waiting on the TX side from 3.12% to 0.3% and the RX side from 4.4%
to 0.3%.
Wait regions are fixed for the lifetime of the ring - once registered,
they are persistent until the ring is torn down. The regions support
minimum wait timeout as well as the regular waits.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-10-22 19:47:00 +00:00
|
|
|
#define IORING_ENTER_EXT_ARG_REG (1U << 6)
|
io_uring: enable toggle of iowait usage when waiting on CQEs
By default, io_uring marks a waiting task as being in iowait, if it's
sleeping waiting on events and there are pending requests. This isn't
necessarily always useful, and may be confusing on non-storage setups
where iowait isn't expected. It can also cause extra power usage, by
preventing the CPU from entering lower sleep states.
This adds a new enter flag, IORING_ENTER_NO_IOWAIT. If set, then
io_uring will not account the sleeping task as being in iowait. If the
kernel supports this feature, then it will be marked by having the
IORING_FEAT_NO_IOWAIT feature flag set.
As the kernel currently does not support separating the iowait
accounting and CPU frequency boosting, the IORING_ENTER_NO_IOWAIT
controls both of these at the same time. In the future, if those do end
up being split, then it'd be possible to control them separately.
However, it seems more likely that the kernel will decouple iowait and
CPU frequency boosting anyway.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2025-03-14 16:19:01 +00:00
|
|
|
#define IORING_ENTER_NO_IOWAIT (1U << 7)
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Passed in for io_uring_setup(2). Copied back with updated info on success
|
|
|
|
|
*/
|
|
|
|
|
struct io_uring_params {
|
|
|
|
|
__u32 sq_entries;
|
|
|
|
|
__u32 cq_entries;
|
|
|
|
|
__u32 flags;
|
io_uring: add submission polling
This enables an application to do IO, without ever entering the kernel.
By using the SQ ring to fill in new sqes and watching for completions
on the CQ ring, we can submit and reap IOs without doing a single system
call. The kernel side thread will poll for new submissions, and in case
of HIPRI/polled IO, it'll also poll for completions.
By default, we allow 1 second of active spinning. This can by changed
by passing in a different grace period at io_uring_register(2) time.
If the thread exceeds this idle time without having any work to do, it
will set:
sq_ring->flags |= IORING_SQ_NEED_WAKEUP.
The application will have to call io_uring_enter() to start things back
up again. If IO is kept busy, that will never be needed. Basically an
application that has this feature enabled will guard it's
io_uring_enter(2) call with:
read_barrier();
if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP)
io_uring_enter(fd, 0, 0, IORING_ENTER_SQ_WAKEUP);
instead of calling it unconditionally.
It's mandatory to use fixed files with this feature. Failure to do so
will result in the application getting an -EBADF CQ entry when
submitting IO.
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-10 18:22:30 +00:00
|
|
|
__u32 sq_thread_cpu;
|
|
|
|
|
__u32 sq_thread_idle;
|
2019-09-06 16:26:21 +00:00
|
|
|
__u32 features;
|
2020-01-28 00:15:48 +00:00
|
|
|
__u32 wq_fd;
|
|
|
|
|
__u32 resv[3];
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
struct io_sqring_offsets sq_off;
|
|
|
|
|
struct io_cqring_offsets cq_off;
|
|
|
|
|
};
|
|
|
|
|
|
2019-09-06 16:26:21 +00:00
|
|
|
/*
|
|
|
|
|
* io_uring_params->features flags
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_FEAT_SINGLE_MMAP (1U << 0)
|
io_uring: add support for backlogged CQ ring
Currently we drop completion events, if the CQ ring is full. That's fine
for requests with bounded completion times, but it may make it harder or
impossible to use io_uring with networked IO where request completion
times are generally unbounded. Or with POLL, for example, which is also
unbounded.
After this patch, we never overflow the ring, we simply store requests
in a backlog for later flushing. This flushing is done automatically by
the kernel. To prevent the backlog from growing indefinitely, if the
backlog is non-empty, we apply back pressure on IO submissions. Any
attempt to submit new IO with a non-empty backlog will get an -EBUSY
return from the kernel. This is a signal to the application that it has
backlogged CQ events, and that it must reap those before being allowed
to submit more IO.
Note that if we do return -EBUSY, we will have filled whatever
backlogged events into the CQ ring first, if there's room. This means
the application can safely reap events WITHOUT entering the kernel and
waiting for them, they are already available in the CQ ring.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-11-06 18:31:17 +00:00
|
|
|
#define IORING_FEAT_NODROP (1U << 1)
|
2019-12-03 01:51:26 +00:00
|
|
|
#define IORING_FEAT_SUBMIT_STABLE (1U << 2)
|
2019-12-25 23:33:42 +00:00
|
|
|
#define IORING_FEAT_RW_CUR_POS (1U << 3)
|
2020-01-27 23:34:48 +00:00
|
|
|
#define IORING_FEAT_CUR_PERSONALITY (1U << 4)
|
io_uring: use poll driven retry for files that support it
Currently io_uring tries any request in a non-blocking manner, if it can,
and then retries from a worker thread if we get -EAGAIN. Now that we have
a new and fancy poll based retry backend, use that to retry requests if
the file supports it.
This means that, for example, an IORING_OP_RECVMSG on a socket no longer
requires an async thread to complete the IO. If we get -EAGAIN reading
from the socket in a non-blocking manner, we arm a poll handler for
notification on when the socket becomes readable. When it does, the
pending read is executed directly by the task again, through the io_uring
task work handlers. Not only is this faster and more efficient, it also
means we're not generating potentially tons of async threads that just
sit and block, waiting for the IO to complete.
The feature is marked with IORING_FEAT_FAST_POLL, meaning that async
pollable IO is fast, and that poll<link>other_op is fast as well.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-02-15 05:23:12 +00:00
|
|
|
#define IORING_FEAT_FAST_POLL (1U << 5)
|
2020-06-17 09:53:55 +00:00
|
|
|
#define IORING_FEAT_POLL_32BITS (1U << 6)
|
2020-09-14 16:51:17 +00:00
|
|
|
#define IORING_FEAT_SQPOLL_NONFIXED (1U << 7)
|
2020-11-03 02:54:37 +00:00
|
|
|
#define IORING_FEAT_EXT_ARG (1U << 8)
|
2021-02-20 18:55:28 +00:00
|
|
|
#define IORING_FEAT_NATIVE_WORKERS (1U << 9)
|
2021-06-10 15:37:38 +00:00
|
|
|
#define IORING_FEAT_RSRC_TAGS (1U << 10)
|
io_uring: add option to skip CQE posting
Emitting a CQE is expensive from the kernel perspective. Often, it's
also not convenient for the userspace, spends some cycles on processing
and just complicates the logic. A similar problems goes for linked
requests, where we post an CQE for each request in the link.
Introduce a new flags, IOSQE_CQE_SKIP_SUCCESS, trying to help with it.
When set and a request completed successfully, it won't generate a CQE.
When fails, it produces an CQE, but all following linked requests will
be CQE-less, regardless whether they have IOSQE_CQE_SKIP_SUCCESS or not.
The notion of "fail" is the same as for link failing-cancellation, where
it's opcode dependent, and _usually_ result >= 0 is a success, but not
always.
Linked timeouts are a bit special. When the requests it's linked to was
not attempted to be executed, e.g. failing linked requests, it follows
the description above. Otherwise, whether a linked timeout will post a
completion or not solely depends on IOSQE_CQE_SKIP_SUCCESS of that
linked timeout request. Linked timeout never "fail" during execution, so
for them it's unconditional. It's expected for users to not really care
about the result of it but rely solely on the result of the master
request. Another reason for such a treatment is that it's racy, and the
timeout callback may be running awhile the master request posts its
completion.
use case 1:
If one doesn't care about results of some requests, e.g. normal
timeouts, just set IOSQE_CQE_SKIP_SUCCESS. Error result will still be
posted and need to be handled.
use case 2:
Set IOSQE_CQE_SKIP_SUCCESS for all requests of a link but the last,
and it'll post a completion only for the last one if everything goes
right, otherwise there will be one only one CQE for the first failed
request.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/0220fbe06f7cf99e6fc71b4297bb1cb6c0e89c2c.1636559119.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-11-10 15:49:32 +00:00
|
|
|
#define IORING_FEAT_CQE_SKIP (1U << 11)
|
2022-04-10 21:13:24 +00:00
|
|
|
#define IORING_FEAT_LINKED_FILE (1U << 12)
|
2023-02-15 00:42:22 +00:00
|
|
|
#define IORING_FEAT_REG_REG_RING (1U << 13)
|
2024-03-05 23:22:04 +00:00
|
|
|
#define IORING_FEAT_RECVSEND_BUNDLE (1U << 14)
|
2024-01-04 17:46:30 +00:00
|
|
|
#define IORING_FEAT_MIN_TIMEOUT (1U << 15)
|
2024-12-05 06:21:09 +00:00
|
|
|
#define IORING_FEAT_RW_ATTR (1U << 16)
|
io_uring: enable toggle of iowait usage when waiting on CQEs
By default, io_uring marks a waiting task as being in iowait, if it's
sleeping waiting on events and there are pending requests. This isn't
necessarily always useful, and may be confusing on non-storage setups
where iowait isn't expected. It can also cause extra power usage, by
preventing the CPU from entering lower sleep states.
This adds a new enter flag, IORING_ENTER_NO_IOWAIT. If set, then
io_uring will not account the sleeping task as being in iowait. If the
kernel supports this feature, then it will be marked by having the
IORING_FEAT_NO_IOWAIT feature flag set.
As the kernel currently does not support separating the iowait
accounting and CPU frequency boosting, the IORING_ENTER_NO_IOWAIT
controls both of these at the same time. In the future, if those do end
up being split, then it'd be possible to control them separately.
However, it seems more likely that the kernel will decouple iowait and
CPU frequency boosting anyway.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2025-03-14 16:19:01 +00:00
|
|
|
#define IORING_FEAT_NO_IOWAIT (1U << 17)
|
2019-09-06 16:26:21 +00:00
|
|
|
|
io_uring: add support for pre-mapped user IO buffers
If we have fixed user buffers, we can map them into the kernel when we
setup the io_uring. That avoids the need to do get_user_pages() for
each and every IO.
To utilize this feature, the application must call io_uring_register()
after having setup an io_uring instance, passing in
IORING_REGISTER_BUFFERS as the opcode. The argument must be a pointer to
an iovec array, and the nr_args should contain how many iovecs the
application wishes to map.
If successful, these buffers are now mapped into the kernel, eligible
for IO. To use these fixed buffers, the application must use the
IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED opcodes, and then
set sqe->index to the desired buffer index. sqe->addr..sqe->addr+seq->len
must point to somewhere inside the indexed buffer.
The application may register buffers throughout the lifetime of the
io_uring instance. It can call io_uring_register() with
IORING_UNREGISTER_BUFFERS as the opcode to unregister the current set of
buffers, and then register a new set. The application need not
unregister buffers explicitly before shutting down the io_uring
instance.
It's perfectly valid to setup a larger buffer, and then sometimes only
use parts of it for an IO. As long as the range is within the originally
mapped region, it will work just fine.
For now, buffers must not be file backed. If file backed buffers are
passed in, the registration will fail with -1/EOPNOTSUPP. This
restriction may be relaxed in the future.
RLIMIT_MEMLOCK is used to check how much memory we can pin. A somewhat
arbitrary 1G per buffer size is also imposed.
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-09 16:16:05 +00:00
|
|
|
/*
|
|
|
|
|
* io_uring_register(2) opcodes and arguments
|
|
|
|
|
*/
|
2024-03-28 21:09:35 +00:00
|
|
|
enum io_uring_register_op {
|
2020-08-27 14:58:29 +00:00
|
|
|
IORING_REGISTER_BUFFERS = 0,
|
|
|
|
|
IORING_UNREGISTER_BUFFERS = 1,
|
|
|
|
|
IORING_REGISTER_FILES = 2,
|
|
|
|
|
IORING_UNREGISTER_FILES = 3,
|
|
|
|
|
IORING_REGISTER_EVENTFD = 4,
|
|
|
|
|
IORING_UNREGISTER_EVENTFD = 5,
|
|
|
|
|
IORING_REGISTER_FILES_UPDATE = 6,
|
|
|
|
|
IORING_REGISTER_EVENTFD_ASYNC = 7,
|
|
|
|
|
IORING_REGISTER_PROBE = 8,
|
|
|
|
|
IORING_REGISTER_PERSONALITY = 9,
|
|
|
|
|
IORING_UNREGISTER_PERSONALITY = 10,
|
2020-08-27 14:58:30 +00:00
|
|
|
IORING_REGISTER_RESTRICTIONS = 11,
|
2020-08-27 14:58:31 +00:00
|
|
|
IORING_REGISTER_ENABLE_RINGS = 12,
|
io_uring: change registration/upd/rsrc tagging ABI
There are ABI moments about recently added rsrc registration/update and
tagging that might become a nuisance in the future. First,
IORING_REGISTER_RSRC[_UPD] hide different types of resources under it,
so breaks fine control over them by restrictions. It works for now, but
once those are wanted under restrictions it would require a rework.
It was also inconvenient trying to fit a new resource not supporting
all the features (e.g. dynamic update) into the interface, so better
to return to IORING_REGISTER_* top level dispatching.
Second, register/update were considered to accept a type of resource,
however that's not a good idea because there might be several ways of
registration of a single resource type, e.g. we may want to add
non-contig buffers or anything more exquisite as dma mapped memory.
So, remove IORING_RSRC_[FILE,BUFFER] out of the ABI, and place them
internally for now to limit changes.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/9b554897a7c17ad6e3becc48dfed2f7af9f423d5.1623339162.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-06-10 15:37:37 +00:00
|
|
|
|
|
|
|
|
/* extended with tagging */
|
|
|
|
|
IORING_REGISTER_FILES2 = 13,
|
|
|
|
|
IORING_REGISTER_FILES_UPDATE2 = 14,
|
|
|
|
|
IORING_REGISTER_BUFFERS2 = 15,
|
|
|
|
|
IORING_REGISTER_BUFFERS_UPDATE = 16,
|
2020-08-27 14:58:29 +00:00
|
|
|
|
2021-06-17 16:19:54 +00:00
|
|
|
/* set/clear io-wq thread affinities */
|
|
|
|
|
IORING_REGISTER_IOWQ_AFF = 17,
|
|
|
|
|
IORING_UNREGISTER_IOWQ_AFF = 18,
|
|
|
|
|
|
2021-09-13 15:44:15 +00:00
|
|
|
/* set/get max number of io-wq workers */
|
2021-08-27 17:33:19 +00:00
|
|
|
IORING_REGISTER_IOWQ_MAX_WORKERS = 19,
|
|
|
|
|
|
io_uring: add support for registering ring file descriptors
Lots of workloads use multiple threads, in which case the file table is
shared between them. This makes getting and putting the ring file
descriptor for each io_uring_enter(2) system call more expensive, as it
involves an atomic get and put for each call.
Similarly to how we allow registering normal file descriptors to avoid
this overhead, add support for an io_uring_register(2) API that allows
to register the ring fds themselves:
1) IORING_REGISTER_RING_FDS - takes an array of io_uring_rsrc_update
structs, and registers them with the task.
2) IORING_UNREGISTER_RING_FDS - takes an array of io_uring_src_update
structs, and unregisters them.
When a ring fd is registered, it is internally represented by an offset.
This offset is returned to the application, and the application then
uses this offset and sets IORING_ENTER_REGISTERED_RING for the
io_uring_enter(2) system call. This works just like using a registered
file descriptor, rather than a real one, in an SQE, where
IOSQE_FIXED_FILE gets set to tell io_uring that we're using an internal
offset/descriptor rather than a real file descriptor.
In initial testing, this provides a nice bump in performance for
threaded applications in real world cases where the batch count (eg
number of requests submitted per io_uring_enter(2) invocation) is low.
In a microbenchmark, submitting NOP requests, we see the following
increases in performance:
Requests per syscall Baseline Registered Increase
----------------------------------------------------------------
1 ~7030K ~8080K +15%
2 ~13120K ~14800K +13%
4 ~22740K ~25300K +11%
Co-developed-by: Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-03-04 15:22:22 +00:00
|
|
|
/* register/unregister io_uring fd with the ring */
|
|
|
|
|
IORING_REGISTER_RING_FDS = 20,
|
|
|
|
|
IORING_UNREGISTER_RING_FDS = 21,
|
|
|
|
|
|
io_uring: add support for ring mapped supplied buffers
Provided buffers allow an application to supply io_uring with buffers
that can then be grabbed for a read/receive request, when the data
source is ready to deliver data. The existing scheme relies on using
IORING_OP_PROVIDE_BUFFERS to do that, but it can be difficult to use
in real world applications. It's pretty efficient if the application
is able to supply back batches of provided buffers when they have been
consumed and the application is ready to recycle them, but if
fragmentation occurs in the buffer space, it can become difficult to
supply enough buffers at the time. This hurts efficiency.
Add a register op, IORING_REGISTER_PBUF_RING, which allows an application
to setup a shared queue for each buffer group of provided buffers. The
application can then supply buffers simply by adding them to this ring,
and the kernel can consume then just as easily. The ring shares the head
with the application, the tail remains private in the kernel.
Provided buffers setup with IORING_REGISTER_PBUF_RING cannot use
IORING_OP_{PROVIDE,REMOVE}_BUFFERS for adding or removing entries to the
ring, they must use the mapped ring. Mapped provided buffer rings can
co-exist with normal provided buffers, just not within the same group ID.
To gauge overhead of the existing scheme and evaluate the mapped ring
approach, a simple NOP benchmark was written. It uses a ring of 128
entries, and submits/completes 32 at the time. 'Replenish' is how
many buffers are provided back at the time after they have been
consumed:
Test Replenish NOPs/sec
================================================================
No provided buffers NA ~30M
Provided buffers 32 ~16M
Provided buffers 1 ~10M
Ring buffers 32 ~27M
Ring buffers 1 ~27M
The ring mapped buffers perform almost as well as not using provided
buffers at all, and they don't care if you provided 1 or more back at
the same time. This means application can just replenish as they go,
rather than need to batch and compact, further reducing overhead in the
application. The NOP benchmark above doesn't need to do any compaction,
so that overhead isn't even reflected in the above test.
Co-developed-by: Dylan Yudaken <dylany@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-04-30 20:38:53 +00:00
|
|
|
/* register ring based provide buffer group */
|
|
|
|
|
IORING_REGISTER_PBUF_RING = 22,
|
|
|
|
|
IORING_UNREGISTER_PBUF_RING = 23,
|
|
|
|
|
|
io_uring: add sync cancelation API through io_uring_register()
The io_uring cancelation API is async, like any other API that we expose
there. For the case of finding a request to cancel, or not finding one,
it is fully sync in that when submission returns, the CQE for both the
cancelation request and the targeted request have been posted to the
CQ ring.
However, if the targeted work is being executed by io-wq, the API can
only start the act of canceling it. This makes it difficult to use in
some circumstances, as the caller then has to wait for the CQEs to come
in and match on the same cancelation data there.
Provide a IORING_REGISTER_SYNC_CANCEL command for io_uring_register()
that does sync cancelations, always. For the io-wq case, it'll wait
for the cancelation to come in before returning. The only expected
returns from this API is:
0 Request found and canceled fine.
> 0 Requests found and canceled. Only happens if asked to
cancel multiple requests, and if the work wasn't in
progress.
-ENOENT Request not found.
-ETIME A timeout on the operation was requested, but the timeout
expired before we could cancel.
and we won't get -EALREADY via this API.
If the timeout value passed in is -1 (tv_sec and tv_nsec), then that
means that no timeout is requested. Otherwise, the timespec passed in
is the amount of time the sync cancel will wait for a successful
cancelation.
Link: https://github.com/axboe/liburing/discussions/608
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-18 16:00:50 +00:00
|
|
|
/* sync cancelation API */
|
|
|
|
|
IORING_REGISTER_SYNC_CANCEL = 24,
|
|
|
|
|
|
2022-06-25 10:55:38 +00:00
|
|
|
/* register a range of fixed file slots for automatic slot allocation */
|
|
|
|
|
IORING_REGISTER_FILE_ALLOC_RANGE = 25,
|
|
|
|
|
|
2023-12-21 16:02:57 +00:00
|
|
|
/* return status information for a buffer group */
|
|
|
|
|
IORING_REGISTER_PBUF_STATUS = 26,
|
|
|
|
|
|
2023-06-08 16:38:38 +00:00
|
|
|
/* set/clear busy poll settings */
|
|
|
|
|
IORING_REGISTER_NAPI = 27,
|
|
|
|
|
IORING_UNREGISTER_NAPI = 28,
|
|
|
|
|
|
2024-08-07 14:18:14 +00:00
|
|
|
IORING_REGISTER_CLOCK = 29,
|
|
|
|
|
|
2024-09-14 14:51:15 +00:00
|
|
|
/* clone registered buffers from source ring to current ring */
|
|
|
|
|
IORING_REGISTER_CLONE_BUFFERS = 30,
|
io_uring: add IORING_REGISTER_COPY_BUFFERS method
Buffers can get registered with io_uring, which allows to skip the
repeated pin_pages, unpin/unref pages for each O_DIRECT operation. This
reduces the overhead of O_DIRECT IO.
However, registrering buffers can take some time. Normally this isn't an
issue as it's done at initialization time (and hence less critical), but
for cases where rings can be created and destroyed as part of an IO
thread pool, registering the same buffers for multiple rings become a
more time sensitive proposition. As an example, let's say an application
has an IO memory pool of 500G. Initial registration takes:
Got 500 huge pages (each 1024MB)
Registered 500 pages in 409 msec
or about 0.4 seconds. If we go higher to 900 1GB huge pages being
registered:
Registered 900 pages in 738 msec
which is, as expected, a fully linear scaling.
Rather than have each ring pin/map/register the same buffer pool,
provide an io_uring_register(2) opcode to simply duplicate the buffers
that are registered with another ring. Adding the same 900GB of
registered buffers to the target ring can then be accomplished in:
Copied 900 pages in 17 usec
While timing differs a bit, this provides around a 25,000-40,000x
speedup for this use case.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-09-11 19:56:08 +00:00
|
|
|
|
io_uring/msg_ring: add support for sending a sync message
Normally MSG_RING requires both a source and a destination ring. But
some users don't always have a ring avilable to send a message from, yet
they still need to notify a target ring.
Add support for using io_uring_register(2) without having a source ring,
using a file descriptor of -1 for that. Internally those are called
blind registration opcodes. Implement IORING_REGISTER_SEND_MSG_RING as a
blind opcode, which simply takes an sqe that the application can put on
the stack and use the normal liburing helpers to initialize it. Then the
app can call:
io_uring_register(-1, IORING_REGISTER_SEND_MSG_RING, &sqe, 1);
and get the same behavior in terms of the target, where a CQE is posted
with the details given in the sqe.
For now this takes a single sqe pointer argument, and hence arg must
be set to that, and nr_args must be 1. Could easily be extended to take
an array of sqes, but for now let's keep it simple.
Link: https://lore.kernel.org/r/20240924115932.116167-3-axboe@kernel.dk
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-09-24 11:57:31 +00:00
|
|
|
/* send MSG_RING without having a ring */
|
|
|
|
|
IORING_REGISTER_SEND_MSG_RING = 31,
|
|
|
|
|
|
2025-02-15 00:09:36 +00:00
|
|
|
/* register a netdev hw rx queue for zerocopy */
|
|
|
|
|
IORING_REGISTER_ZCRX_IFQ = 32,
|
io_uring/register: add IORING_REGISTER_RESIZE_RINGS
Once a ring has been created, the size of the CQ and SQ rings are fixed.
Usually this isn't a problem on the SQ ring side, as it merely controls
the available number of requests that can be submitted in a single
system call, and there's rarely a need to change that.
For the CQ ring, it's a different story. For most efficient use of
io_uring, it's important that the CQ ring never overflows. This means
that applications must size it for the worst case scenario, which can
be wasteful.
Add IORING_REGISTER_RESIZE_RINGS, which allows an application to resize
the existing rings. It takes a struct io_uring_params argument, the same
one which is used to setup the ring initially, and resizes rings
according to the sizes given.
Certain properties are always inherited from the original ring setup,
like SQE128/CQE32 and other setup options. The implementation only
allows flag associated with how the CQ ring is sized and clamped.
Existing unconsumed SQE and CQE entries are copied as part of the
process. If either the SQ or CQ resized destination ring cannot hold the
entries already present in the source rings, then the operation is failed
with -EOVERFLOW. Any register op holds ->uring_lock, which prevents new
submissions, and the internal mapping holds the completion lock as well
across moving CQ ring state.
To prevent races between mmap and ring resizing, add a mutex that's
solely used to serialize ring resize and mmap. mmap_sem can't be used
here, as as fork'ed process may be doing mmaps on the ring as well.
The ctx->resize_lock is held across mmap operations, and the resize
will grab it before swapping out the already mapped new data.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-10-21 19:34:10 +00:00
|
|
|
|
|
|
|
|
/* resize CQ ring */
|
|
|
|
|
IORING_REGISTER_RESIZE_RINGS = 33,
|
|
|
|
|
|
2024-11-15 16:54:42 +00:00
|
|
|
IORING_REGISTER_MEM_REGION = 34,
|
|
|
|
|
|
2020-08-27 14:58:29 +00:00
|
|
|
/* this goes last */
|
2023-02-15 00:42:22 +00:00
|
|
|
IORING_REGISTER_LAST,
|
|
|
|
|
|
|
|
|
|
/* flag added to the opcode to use a registered ring fd */
|
|
|
|
|
IORING_REGISTER_USE_REGISTERED_RING = 1U << 31
|
2020-08-27 14:58:29 +00:00
|
|
|
};
|
2019-10-03 19:59:56 +00:00
|
|
|
|
2021-09-13 15:44:15 +00:00
|
|
|
/* io-wq worker categories */
|
2024-03-28 21:09:35 +00:00
|
|
|
enum io_wq_type {
|
2021-09-13 15:44:15 +00:00
|
|
|
IO_WQ_BOUND,
|
|
|
|
|
IO_WQ_UNBOUND,
|
|
|
|
|
};
|
|
|
|
|
|
2021-01-15 17:37:44 +00:00
|
|
|
/* deprecated, see struct io_uring_rsrc_update */
|
2019-10-03 19:59:56 +00:00
|
|
|
struct io_uring_files_update {
|
|
|
|
|
__u32 offset;
|
2020-01-15 16:35:38 +00:00
|
|
|
__u32 resv;
|
|
|
|
|
__aligned_u64 /* __s32 * */ fds;
|
2019-10-03 19:59:56 +00:00
|
|
|
};
|
io_uring: add support for pre-mapped user IO buffers
If we have fixed user buffers, we can map them into the kernel when we
setup the io_uring. That avoids the need to do get_user_pages() for
each and every IO.
To utilize this feature, the application must call io_uring_register()
after having setup an io_uring instance, passing in
IORING_REGISTER_BUFFERS as the opcode. The argument must be a pointer to
an iovec array, and the nr_args should contain how many iovecs the
application wishes to map.
If successful, these buffers are now mapped into the kernel, eligible
for IO. To use these fixed buffers, the application must use the
IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED opcodes, and then
set sqe->index to the desired buffer index. sqe->addr..sqe->addr+seq->len
must point to somewhere inside the indexed buffer.
The application may register buffers throughout the lifetime of the
io_uring instance. It can call io_uring_register() with
IORING_UNREGISTER_BUFFERS as the opcode to unregister the current set of
buffers, and then register a new set. The application need not
unregister buffers explicitly before shutting down the io_uring
instance.
It's perfectly valid to setup a larger buffer, and then sometimes only
use parts of it for an IO. As long as the range is within the originally
mapped region, it will work just fine.
For now, buffers must not be file backed. If file backed buffers are
passed in, the registration will fail with -1/EOPNOTSUPP. This
restriction may be relaxed in the future.
RLIMIT_MEMLOCK is used to check how much memory we can pin. A somewhat
arbitrary 1G per buffer size is also imposed.
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-09 16:16:05 +00:00
|
|
|
|
io_uring: introduce concept of memory regions
We've got a good number of mappings we share with the userspace, that
includes the main rings, provided buffer rings, upcoming rings for
zerocopy rx and more. All of them duplicate user argument parsing and
some internal details as well (page pinnning, huge page optimisations,
mmap'ing, etc.)
Introduce a notion of regions. For userspace for now it's just a new
structure called struct io_uring_region_desc which is supposed to
parameterise all such mapping / queue creations. A region either
represents a user provided chunk of memory, in which case the user_addr
field should point to it, or a request for the kernel to allocate the
memory, in which case the user would need to mmap it after using the
offset returned in the mmap_offset field. With a uniform userspace API
we can avoid additional boiler plate code and apply future optimisation
to all of them at once.
Internally, there is a new structure struct io_mapped_region holding all
relevant runtime information and some helpers to work with it. This
patch limits it to user provided regions.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/0e6fe25818dfbaebd1bd90b870a6cac503fe1a24.1731689588.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-11-15 16:54:41 +00:00
|
|
|
enum {
|
|
|
|
|
/* initialise with user provided memory pointed by user_addr */
|
|
|
|
|
IORING_MEM_REGION_TYPE_USER = 1,
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct io_uring_region_desc {
|
|
|
|
|
__u64 user_addr;
|
|
|
|
|
__u64 size;
|
|
|
|
|
__u32 flags;
|
|
|
|
|
__u32 id;
|
|
|
|
|
__u64 mmap_offset;
|
|
|
|
|
__u64 __resv[4];
|
|
|
|
|
};
|
|
|
|
|
|
2024-11-15 16:54:43 +00:00
|
|
|
enum {
|
|
|
|
|
/* expose the region as registered wait arguments */
|
|
|
|
|
IORING_MEM_REGION_REG_WAIT_ARG = 1,
|
|
|
|
|
};
|
|
|
|
|
|
2024-11-15 16:54:42 +00:00
|
|
|
struct io_uring_mem_region_reg {
|
|
|
|
|
__u64 region_uptr; /* struct io_uring_region_desc * */
|
|
|
|
|
__u64 flags;
|
|
|
|
|
__u64 __resv[2];
|
|
|
|
|
};
|
|
|
|
|
|
2022-05-09 15:29:14 +00:00
|
|
|
/*
|
|
|
|
|
* Register a fully sparse file space, rather than pass in an array of all
|
|
|
|
|
* -1 file descriptors.
|
|
|
|
|
*/
|
|
|
|
|
#define IORING_RSRC_REGISTER_SPARSE (1U << 0)
|
|
|
|
|
|
2021-04-25 13:32:21 +00:00
|
|
|
struct io_uring_rsrc_register {
|
|
|
|
|
__u32 nr;
|
2022-05-09 15:29:14 +00:00
|
|
|
__u32 flags;
|
io_uring: change registration/upd/rsrc tagging ABI
There are ABI moments about recently added rsrc registration/update and
tagging that might become a nuisance in the future. First,
IORING_REGISTER_RSRC[_UPD] hide different types of resources under it,
so breaks fine control over them by restrictions. It works for now, but
once those are wanted under restrictions it would require a rework.
It was also inconvenient trying to fit a new resource not supporting
all the features (e.g. dynamic update) into the interface, so better
to return to IORING_REGISTER_* top level dispatching.
Second, register/update were considered to accept a type of resource,
however that's not a good idea because there might be several ways of
registration of a single resource type, e.g. we may want to add
non-contig buffers or anything more exquisite as dma mapped memory.
So, remove IORING_RSRC_[FILE,BUFFER] out of the ABI, and place them
internally for now to limit changes.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/9b554897a7c17ad6e3becc48dfed2f7af9f423d5.1623339162.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-06-10 15:37:37 +00:00
|
|
|
__u64 resv2;
|
2021-04-25 13:32:21 +00:00
|
|
|
__aligned_u64 data;
|
|
|
|
|
__aligned_u64 tags;
|
|
|
|
|
};
|
|
|
|
|
|
2021-04-25 13:32:22 +00:00
|
|
|
struct io_uring_rsrc_update {
|
|
|
|
|
__u32 offset;
|
|
|
|
|
__u32 resv;
|
|
|
|
|
__aligned_u64 data;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct io_uring_rsrc_update2 {
|
|
|
|
|
__u32 offset;
|
|
|
|
|
__u32 resv;
|
|
|
|
|
__aligned_u64 data;
|
|
|
|
|
__aligned_u64 tags;
|
|
|
|
|
__u32 nr;
|
io_uring: change registration/upd/rsrc tagging ABI
There are ABI moments about recently added rsrc registration/update and
tagging that might become a nuisance in the future. First,
IORING_REGISTER_RSRC[_UPD] hide different types of resources under it,
so breaks fine control over them by restrictions. It works for now, but
once those are wanted under restrictions it would require a rework.
It was also inconvenient trying to fit a new resource not supporting
all the features (e.g. dynamic update) into the interface, so better
to return to IORING_REGISTER_* top level dispatching.
Second, register/update were considered to accept a type of resource,
however that's not a good idea because there might be several ways of
registration of a single resource type, e.g. we may want to add
non-contig buffers or anything more exquisite as dma mapped memory.
So, remove IORING_RSRC_[FILE,BUFFER] out of the ABI, and place them
internally for now to limit changes.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/9b554897a7c17ad6e3becc48dfed2f7af9f423d5.1623339162.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-06-10 15:37:37 +00:00
|
|
|
__u32 resv2;
|
2021-04-25 13:32:22 +00:00
|
|
|
};
|
|
|
|
|
|
2021-01-26 20:23:28 +00:00
|
|
|
/* Skip updating fd indexes set to this value in the fd table */
|
|
|
|
|
#define IORING_REGISTER_FILES_SKIP (-2)
|
|
|
|
|
|
2020-01-16 22:36:52 +00:00
|
|
|
#define IO_URING_OP_SUPPORTED (1U << 0)
|
|
|
|
|
|
|
|
|
|
struct io_uring_probe_op {
|
|
|
|
|
__u8 op;
|
|
|
|
|
__u8 resv;
|
|
|
|
|
__u16 flags; /* IO_URING_OP_* flags */
|
|
|
|
|
__u32 resv2;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct io_uring_probe {
|
|
|
|
|
__u8 last_op; /* last opcode supported */
|
|
|
|
|
__u8 ops_len; /* length of ops[] array below */
|
|
|
|
|
__u16 resv;
|
|
|
|
|
__u32 resv2[3];
|
2022-06-28 19:33:20 +00:00
|
|
|
struct io_uring_probe_op ops[];
|
2020-01-16 22:36:52 +00:00
|
|
|
};
|
|
|
|
|
|
2020-08-27 14:58:30 +00:00
|
|
|
struct io_uring_restriction {
|
|
|
|
|
__u16 opcode;
|
|
|
|
|
union {
|
|
|
|
|
__u8 register_op; /* IORING_RESTRICTION_REGISTER_OP */
|
|
|
|
|
__u8 sqe_op; /* IORING_RESTRICTION_SQE_OP */
|
|
|
|
|
__u8 sqe_flags; /* IORING_RESTRICTION_SQE_FLAGS_* */
|
|
|
|
|
};
|
|
|
|
|
__u8 resv;
|
|
|
|
|
__u32 resv2[3];
|
|
|
|
|
};
|
|
|
|
|
|
2024-08-07 14:18:14 +00:00
|
|
|
struct io_uring_clock_register {
|
|
|
|
|
__u32 clockid;
|
|
|
|
|
__u32 __resv[3];
|
|
|
|
|
};
|
|
|
|
|
|
io_uring: add IORING_REGISTER_COPY_BUFFERS method
Buffers can get registered with io_uring, which allows to skip the
repeated pin_pages, unpin/unref pages for each O_DIRECT operation. This
reduces the overhead of O_DIRECT IO.
However, registrering buffers can take some time. Normally this isn't an
issue as it's done at initialization time (and hence less critical), but
for cases where rings can be created and destroyed as part of an IO
thread pool, registering the same buffers for multiple rings become a
more time sensitive proposition. As an example, let's say an application
has an IO memory pool of 500G. Initial registration takes:
Got 500 huge pages (each 1024MB)
Registered 500 pages in 409 msec
or about 0.4 seconds. If we go higher to 900 1GB huge pages being
registered:
Registered 900 pages in 738 msec
which is, as expected, a fully linear scaling.
Rather than have each ring pin/map/register the same buffer pool,
provide an io_uring_register(2) opcode to simply duplicate the buffers
that are registered with another ring. Adding the same 900GB of
registered buffers to the target ring can then be accomplished in:
Copied 900 pages in 17 usec
While timing differs a bit, this provides around a 25,000-40,000x
speedup for this use case.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-09-11 19:56:08 +00:00
|
|
|
enum {
|
2024-10-29 13:50:56 +00:00
|
|
|
IORING_REGISTER_SRC_REGISTERED = (1U << 0),
|
|
|
|
|
IORING_REGISTER_DST_REPLACE = (1U << 1),
|
io_uring: add IORING_REGISTER_COPY_BUFFERS method
Buffers can get registered with io_uring, which allows to skip the
repeated pin_pages, unpin/unref pages for each O_DIRECT operation. This
reduces the overhead of O_DIRECT IO.
However, registrering buffers can take some time. Normally this isn't an
issue as it's done at initialization time (and hence less critical), but
for cases where rings can be created and destroyed as part of an IO
thread pool, registering the same buffers for multiple rings become a
more time sensitive proposition. As an example, let's say an application
has an IO memory pool of 500G. Initial registration takes:
Got 500 huge pages (each 1024MB)
Registered 500 pages in 409 msec
or about 0.4 seconds. If we go higher to 900 1GB huge pages being
registered:
Registered 900 pages in 738 msec
which is, as expected, a fully linear scaling.
Rather than have each ring pin/map/register the same buffer pool,
provide an io_uring_register(2) opcode to simply duplicate the buffers
that are registered with another ring. Adding the same 900GB of
registered buffers to the target ring can then be accomplished in:
Copied 900 pages in 17 usec
While timing differs a bit, this provides around a 25,000-40,000x
speedup for this use case.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-09-11 19:56:08 +00:00
|
|
|
};
|
|
|
|
|
|
2024-09-14 14:51:15 +00:00
|
|
|
struct io_uring_clone_buffers {
|
io_uring: add IORING_REGISTER_COPY_BUFFERS method
Buffers can get registered with io_uring, which allows to skip the
repeated pin_pages, unpin/unref pages for each O_DIRECT operation. This
reduces the overhead of O_DIRECT IO.
However, registrering buffers can take some time. Normally this isn't an
issue as it's done at initialization time (and hence less critical), but
for cases where rings can be created and destroyed as part of an IO
thread pool, registering the same buffers for multiple rings become a
more time sensitive proposition. As an example, let's say an application
has an IO memory pool of 500G. Initial registration takes:
Got 500 huge pages (each 1024MB)
Registered 500 pages in 409 msec
or about 0.4 seconds. If we go higher to 900 1GB huge pages being
registered:
Registered 900 pages in 738 msec
which is, as expected, a fully linear scaling.
Rather than have each ring pin/map/register the same buffer pool,
provide an io_uring_register(2) opcode to simply duplicate the buffers
that are registered with another ring. Adding the same 900GB of
registered buffers to the target ring can then be accomplished in:
Copied 900 pages in 17 usec
While timing differs a bit, this provides around a 25,000-40,000x
speedup for this use case.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-09-11 19:56:08 +00:00
|
|
|
__u32 src_fd;
|
|
|
|
|
__u32 flags;
|
2024-10-29 00:43:13 +00:00
|
|
|
__u32 src_off;
|
|
|
|
|
__u32 dst_off;
|
|
|
|
|
__u32 nr;
|
|
|
|
|
__u32 pad[3];
|
io_uring: add IORING_REGISTER_COPY_BUFFERS method
Buffers can get registered with io_uring, which allows to skip the
repeated pin_pages, unpin/unref pages for each O_DIRECT operation. This
reduces the overhead of O_DIRECT IO.
However, registrering buffers can take some time. Normally this isn't an
issue as it's done at initialization time (and hence less critical), but
for cases where rings can be created and destroyed as part of an IO
thread pool, registering the same buffers for multiple rings become a
more time sensitive proposition. As an example, let's say an application
has an IO memory pool of 500G. Initial registration takes:
Got 500 huge pages (each 1024MB)
Registered 500 pages in 409 msec
or about 0.4 seconds. If we go higher to 900 1GB huge pages being
registered:
Registered 900 pages in 738 msec
which is, as expected, a fully linear scaling.
Rather than have each ring pin/map/register the same buffer pool,
provide an io_uring_register(2) opcode to simply duplicate the buffers
that are registered with another ring. Adding the same 900GB of
registered buffers to the target ring can then be accomplished in:
Copied 900 pages in 17 usec
While timing differs a bit, this provides around a 25,000-40,000x
speedup for this use case.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-09-11 19:56:08 +00:00
|
|
|
};
|
|
|
|
|
|
io_uring: add support for ring mapped supplied buffers
Provided buffers allow an application to supply io_uring with buffers
that can then be grabbed for a read/receive request, when the data
source is ready to deliver data. The existing scheme relies on using
IORING_OP_PROVIDE_BUFFERS to do that, but it can be difficult to use
in real world applications. It's pretty efficient if the application
is able to supply back batches of provided buffers when they have been
consumed and the application is ready to recycle them, but if
fragmentation occurs in the buffer space, it can become difficult to
supply enough buffers at the time. This hurts efficiency.
Add a register op, IORING_REGISTER_PBUF_RING, which allows an application
to setup a shared queue for each buffer group of provided buffers. The
application can then supply buffers simply by adding them to this ring,
and the kernel can consume then just as easily. The ring shares the head
with the application, the tail remains private in the kernel.
Provided buffers setup with IORING_REGISTER_PBUF_RING cannot use
IORING_OP_{PROVIDE,REMOVE}_BUFFERS for adding or removing entries to the
ring, they must use the mapped ring. Mapped provided buffer rings can
co-exist with normal provided buffers, just not within the same group ID.
To gauge overhead of the existing scheme and evaluate the mapped ring
approach, a simple NOP benchmark was written. It uses a ring of 128
entries, and submits/completes 32 at the time. 'Replenish' is how
many buffers are provided back at the time after they have been
consumed:
Test Replenish NOPs/sec
================================================================
No provided buffers NA ~30M
Provided buffers 32 ~16M
Provided buffers 1 ~10M
Ring buffers 32 ~27M
Ring buffers 1 ~27M
The ring mapped buffers perform almost as well as not using provided
buffers at all, and they don't care if you provided 1 or more back at
the same time. This means application can just replenish as they go,
rather than need to batch and compact, further reducing overhead in the
application. The NOP benchmark above doesn't need to do any compaction,
so that overhead isn't even reflected in the above test.
Co-developed-by: Dylan Yudaken <dylany@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-04-30 20:38:53 +00:00
|
|
|
struct io_uring_buf {
|
|
|
|
|
__u64 addr;
|
|
|
|
|
__u32 len;
|
|
|
|
|
__u16 bid;
|
|
|
|
|
__u16 resv;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct io_uring_buf_ring {
|
|
|
|
|
union {
|
|
|
|
|
/*
|
|
|
|
|
* To avoid spilling into more pages than we need to, the
|
|
|
|
|
* ring tail is overlaid with the io_uring_buf->resv field.
|
|
|
|
|
*/
|
|
|
|
|
struct {
|
|
|
|
|
__u64 resv1;
|
|
|
|
|
__u32 resv2;
|
|
|
|
|
__u16 resv3;
|
|
|
|
|
__u16 tail;
|
|
|
|
|
};
|
2023-01-05 19:05:11 +00:00
|
|
|
__DECLARE_FLEX_ARRAY(struct io_uring_buf, bufs);
|
io_uring: add support for ring mapped supplied buffers
Provided buffers allow an application to supply io_uring with buffers
that can then be grabbed for a read/receive request, when the data
source is ready to deliver data. The existing scheme relies on using
IORING_OP_PROVIDE_BUFFERS to do that, but it can be difficult to use
in real world applications. It's pretty efficient if the application
is able to supply back batches of provided buffers when they have been
consumed and the application is ready to recycle them, but if
fragmentation occurs in the buffer space, it can become difficult to
supply enough buffers at the time. This hurts efficiency.
Add a register op, IORING_REGISTER_PBUF_RING, which allows an application
to setup a shared queue for each buffer group of provided buffers. The
application can then supply buffers simply by adding them to this ring,
and the kernel can consume then just as easily. The ring shares the head
with the application, the tail remains private in the kernel.
Provided buffers setup with IORING_REGISTER_PBUF_RING cannot use
IORING_OP_{PROVIDE,REMOVE}_BUFFERS for adding or removing entries to the
ring, they must use the mapped ring. Mapped provided buffer rings can
co-exist with normal provided buffers, just not within the same group ID.
To gauge overhead of the existing scheme and evaluate the mapped ring
approach, a simple NOP benchmark was written. It uses a ring of 128
entries, and submits/completes 32 at the time. 'Replenish' is how
many buffers are provided back at the time after they have been
consumed:
Test Replenish NOPs/sec
================================================================
No provided buffers NA ~30M
Provided buffers 32 ~16M
Provided buffers 1 ~10M
Ring buffers 32 ~27M
Ring buffers 1 ~27M
The ring mapped buffers perform almost as well as not using provided
buffers at all, and they don't care if you provided 1 or more back at
the same time. This means application can just replenish as they go,
rather than need to batch and compact, further reducing overhead in the
application. The NOP benchmark above doesn't need to do any compaction,
so that overhead isn't even reflected in the above test.
Co-developed-by: Dylan Yudaken <dylany@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-04-30 20:38:53 +00:00
|
|
|
};
|
|
|
|
|
};
|
|
|
|
|
|
2023-03-14 17:07:19 +00:00
|
|
|
/*
|
|
|
|
|
* Flags for IORING_REGISTER_PBUF_RING.
|
|
|
|
|
*
|
|
|
|
|
* IOU_PBUF_RING_MMAP: If set, kernel will allocate the memory for the ring.
|
|
|
|
|
* The application must not set a ring_addr in struct
|
|
|
|
|
* io_uring_buf_reg, instead it must subsequently call
|
|
|
|
|
* mmap(2) with the offset set as:
|
|
|
|
|
* IORING_OFF_PBUF_RING | (bgid << IORING_OFF_PBUF_SHIFT)
|
|
|
|
|
* to get a virtual mapping for the ring.
|
io_uring/kbuf: add support for incremental buffer consumption
By default, any recv/read operation that uses provided buffers will
consume at least 1 buffer fully (and maybe more, in case of bundles).
This adds support for incremental consumption, meaning that an
application may add large buffers, and each read/recv will just consume
the part of the buffer that it needs.
For example, let's say an application registers 1MB buffers in a
provided buffer ring, for streaming receives. If it gets a short recv,
then the full 1MB buffer will be consumed and passed back to the
application. With incremental consumption, only the part that was
actually used is consumed, and the buffer remains the current one.
This means that both the application and the kernel needs to keep track
of what the current receive point is. Each recv will still pass back a
buffer ID and the size consumed, the only difference is that before the
next receive would always be the next buffer in the ring. Now the same
buffer ID may return multiple receives, each at an offset into that
buffer from where the previous receive left off. Example:
Application registers a provided buffer ring, and adds two 32K buffers
to the ring.
Buffer1 address: 0x1000000 (buffer ID 0)
Buffer2 address: 0x2000000 (buffer ID 1)
A recv completion is received with the following values:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
and the application now knows that 4096b of data is available at
0x1000000, the start of that buffer, and that more data from this buffer
will be coming. Now the next receive comes in:
cqe->res 0x2010 (8k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
which tells the application that 8k is available where the last
completion left off, at 0x1001000. Next completion is:
cqe->res 0x5000 (20k bytes received)
cqe->flags 0x1 (CQE_F_BUFFER set, buffer ID 0)
and the application now knows that 20k of data is available at
0x1003000, which is where the previous receive ended. CQE_F_BUF_MORE
isn't set, as no more data is available in this buffer ID. The next
completion is then:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x10001 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 1)
which tells the application that buffer ID 1 is now the current one,
hence there's 4k of valid data at 0x2000000. 0x2001000 will be the next
receive point for this buffer ID.
When a buffer will be reused by future CQE completions,
IORING_CQE_BUF_MORE will be set in cqe->flags. This tells the application
that the kernel isn't done with the buffer yet, and that it should expect
more completions for this buffer ID. Will only be set by provided buffer
rings setup with IOU_PBUF_RING INC, as that's the only type of buffer
that will see multiple consecutive completions for the same buffer ID.
For any other provided buffer type, any completion that passes back
a buffer to the application is final.
Once a buffer has been fully consumed, the buffer ring head is
incremented and the next receive will indicate the next buffer ID in the
CQE cflags.
On the send side, the application can manage how much data is sent from
an existing buffer by setting sqe->len to the desired send length.
An application can request incremental consumption by setting
IOU_PBUF_RING_INC in the provided buffer ring registration. Outside of
that, any provided buffer ring setup and buffer additions is done like
before, no changes there. The only change is in how an application may
see multiple completions for the same buffer ID, hence needing to know
where the next receive will happen.
Note that like existing provided buffer rings, this should not be used
with IOSQE_ASYNC, as both really require the ring to remain locked over
the duration of the buffer selection and the operation completion. It
will consume a buffer otherwise regardless of the size of the IO done.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-08-09 17:20:45 +00:00
|
|
|
* IOU_PBUF_RING_INC: If set, buffers consumed from this buffer ring can be
|
|
|
|
|
* consumed incrementally. Normally one (or more) buffers
|
|
|
|
|
* are fully consumed. With incremental consumptions, it's
|
|
|
|
|
* feasible to register big ranges of buffers, and each
|
|
|
|
|
* use of it will consume only as much as it needs. This
|
|
|
|
|
* requires that both the kernel and application keep
|
|
|
|
|
* track of where the current read/recv index is at.
|
2023-03-14 17:07:19 +00:00
|
|
|
*/
|
2024-03-28 21:09:35 +00:00
|
|
|
enum io_uring_register_pbuf_ring_flags {
|
2023-03-14 17:07:19 +00:00
|
|
|
IOU_PBUF_RING_MMAP = 1,
|
io_uring/kbuf: add support for incremental buffer consumption
By default, any recv/read operation that uses provided buffers will
consume at least 1 buffer fully (and maybe more, in case of bundles).
This adds support for incremental consumption, meaning that an
application may add large buffers, and each read/recv will just consume
the part of the buffer that it needs.
For example, let's say an application registers 1MB buffers in a
provided buffer ring, for streaming receives. If it gets a short recv,
then the full 1MB buffer will be consumed and passed back to the
application. With incremental consumption, only the part that was
actually used is consumed, and the buffer remains the current one.
This means that both the application and the kernel needs to keep track
of what the current receive point is. Each recv will still pass back a
buffer ID and the size consumed, the only difference is that before the
next receive would always be the next buffer in the ring. Now the same
buffer ID may return multiple receives, each at an offset into that
buffer from where the previous receive left off. Example:
Application registers a provided buffer ring, and adds two 32K buffers
to the ring.
Buffer1 address: 0x1000000 (buffer ID 0)
Buffer2 address: 0x2000000 (buffer ID 1)
A recv completion is received with the following values:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
and the application now knows that 4096b of data is available at
0x1000000, the start of that buffer, and that more data from this buffer
will be coming. Now the next receive comes in:
cqe->res 0x2010 (8k bytes received)
cqe->flags 0x11 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 0)
which tells the application that 8k is available where the last
completion left off, at 0x1001000. Next completion is:
cqe->res 0x5000 (20k bytes received)
cqe->flags 0x1 (CQE_F_BUFFER set, buffer ID 0)
and the application now knows that 20k of data is available at
0x1003000, which is where the previous receive ended. CQE_F_BUF_MORE
isn't set, as no more data is available in this buffer ID. The next
completion is then:
cqe->res 0x1000 (4k bytes received)
cqe->flags 0x10001 (CQE_F_BUFFER|CQE_F_BUF_MORE set, buffer ID 1)
which tells the application that buffer ID 1 is now the current one,
hence there's 4k of valid data at 0x2000000. 0x2001000 will be the next
receive point for this buffer ID.
When a buffer will be reused by future CQE completions,
IORING_CQE_BUF_MORE will be set in cqe->flags. This tells the application
that the kernel isn't done with the buffer yet, and that it should expect
more completions for this buffer ID. Will only be set by provided buffer
rings setup with IOU_PBUF_RING INC, as that's the only type of buffer
that will see multiple consecutive completions for the same buffer ID.
For any other provided buffer type, any completion that passes back
a buffer to the application is final.
Once a buffer has been fully consumed, the buffer ring head is
incremented and the next receive will indicate the next buffer ID in the
CQE cflags.
On the send side, the application can manage how much data is sent from
an existing buffer by setting sqe->len to the desired send length.
An application can request incremental consumption by setting
IOU_PBUF_RING_INC in the provided buffer ring registration. Outside of
that, any provided buffer ring setup and buffer additions is done like
before, no changes there. The only change is in how an application may
see multiple completions for the same buffer ID, hence needing to know
where the next receive will happen.
Note that like existing provided buffer rings, this should not be used
with IOSQE_ASYNC, as both really require the ring to remain locked over
the duration of the buffer selection and the operation completion. It
will consume a buffer otherwise regardless of the size of the IO done.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-08-09 17:20:45 +00:00
|
|
|
IOU_PBUF_RING_INC = 2,
|
2023-03-14 17:07:19 +00:00
|
|
|
};
|
|
|
|
|
|
io_uring: add support for ring mapped supplied buffers
Provided buffers allow an application to supply io_uring with buffers
that can then be grabbed for a read/receive request, when the data
source is ready to deliver data. The existing scheme relies on using
IORING_OP_PROVIDE_BUFFERS to do that, but it can be difficult to use
in real world applications. It's pretty efficient if the application
is able to supply back batches of provided buffers when they have been
consumed and the application is ready to recycle them, but if
fragmentation occurs in the buffer space, it can become difficult to
supply enough buffers at the time. This hurts efficiency.
Add a register op, IORING_REGISTER_PBUF_RING, which allows an application
to setup a shared queue for each buffer group of provided buffers. The
application can then supply buffers simply by adding them to this ring,
and the kernel can consume then just as easily. The ring shares the head
with the application, the tail remains private in the kernel.
Provided buffers setup with IORING_REGISTER_PBUF_RING cannot use
IORING_OP_{PROVIDE,REMOVE}_BUFFERS for adding or removing entries to the
ring, they must use the mapped ring. Mapped provided buffer rings can
co-exist with normal provided buffers, just not within the same group ID.
To gauge overhead of the existing scheme and evaluate the mapped ring
approach, a simple NOP benchmark was written. It uses a ring of 128
entries, and submits/completes 32 at the time. 'Replenish' is how
many buffers are provided back at the time after they have been
consumed:
Test Replenish NOPs/sec
================================================================
No provided buffers NA ~30M
Provided buffers 32 ~16M
Provided buffers 1 ~10M
Ring buffers 32 ~27M
Ring buffers 1 ~27M
The ring mapped buffers perform almost as well as not using provided
buffers at all, and they don't care if you provided 1 or more back at
the same time. This means application can just replenish as they go,
rather than need to batch and compact, further reducing overhead in the
application. The NOP benchmark above doesn't need to do any compaction,
so that overhead isn't even reflected in the above test.
Co-developed-by: Dylan Yudaken <dylany@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-04-30 20:38:53 +00:00
|
|
|
/* argument for IORING_(UN)REGISTER_PBUF_RING */
|
|
|
|
|
struct io_uring_buf_reg {
|
|
|
|
|
__u64 ring_addr;
|
|
|
|
|
__u32 ring_entries;
|
|
|
|
|
__u16 bgid;
|
2023-03-14 17:01:45 +00:00
|
|
|
__u16 flags;
|
io_uring: add support for ring mapped supplied buffers
Provided buffers allow an application to supply io_uring with buffers
that can then be grabbed for a read/receive request, when the data
source is ready to deliver data. The existing scheme relies on using
IORING_OP_PROVIDE_BUFFERS to do that, but it can be difficult to use
in real world applications. It's pretty efficient if the application
is able to supply back batches of provided buffers when they have been
consumed and the application is ready to recycle them, but if
fragmentation occurs in the buffer space, it can become difficult to
supply enough buffers at the time. This hurts efficiency.
Add a register op, IORING_REGISTER_PBUF_RING, which allows an application
to setup a shared queue for each buffer group of provided buffers. The
application can then supply buffers simply by adding them to this ring,
and the kernel can consume then just as easily. The ring shares the head
with the application, the tail remains private in the kernel.
Provided buffers setup with IORING_REGISTER_PBUF_RING cannot use
IORING_OP_{PROVIDE,REMOVE}_BUFFERS for adding or removing entries to the
ring, they must use the mapped ring. Mapped provided buffer rings can
co-exist with normal provided buffers, just not within the same group ID.
To gauge overhead of the existing scheme and evaluate the mapped ring
approach, a simple NOP benchmark was written. It uses a ring of 128
entries, and submits/completes 32 at the time. 'Replenish' is how
many buffers are provided back at the time after they have been
consumed:
Test Replenish NOPs/sec
================================================================
No provided buffers NA ~30M
Provided buffers 32 ~16M
Provided buffers 1 ~10M
Ring buffers 32 ~27M
Ring buffers 1 ~27M
The ring mapped buffers perform almost as well as not using provided
buffers at all, and they don't care if you provided 1 or more back at
the same time. This means application can just replenish as they go,
rather than need to batch and compact, further reducing overhead in the
application. The NOP benchmark above doesn't need to do any compaction,
so that overhead isn't even reflected in the above test.
Co-developed-by: Dylan Yudaken <dylany@fb.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-04-30 20:38:53 +00:00
|
|
|
__u64 resv[3];
|
|
|
|
|
};
|
|
|
|
|
|
2023-12-21 16:02:57 +00:00
|
|
|
/* argument for IORING_REGISTER_PBUF_STATUS */
|
|
|
|
|
struct io_uring_buf_status {
|
|
|
|
|
__u32 buf_group; /* input */
|
|
|
|
|
__u32 head; /* output */
|
|
|
|
|
__u32 resv[8];
|
|
|
|
|
};
|
|
|
|
|
|
2024-10-13 18:29:24 +00:00
|
|
|
enum io_uring_napi_op {
|
|
|
|
|
/* register/ungister backward compatible opcode */
|
|
|
|
|
IO_URING_NAPI_REGISTER_OP = 0,
|
|
|
|
|
|
|
|
|
|
/* opcodes to update napi_list when static tracking is used */
|
|
|
|
|
IO_URING_NAPI_STATIC_ADD_ID = 1,
|
|
|
|
|
IO_URING_NAPI_STATIC_DEL_ID = 2
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
enum io_uring_napi_tracking_strategy {
|
|
|
|
|
/* value must be 0 for backward compatibility */
|
|
|
|
|
IO_URING_NAPI_TRACKING_DYNAMIC = 0,
|
|
|
|
|
IO_URING_NAPI_TRACKING_STATIC = 1,
|
|
|
|
|
IO_URING_NAPI_TRACKING_INACTIVE = 255
|
|
|
|
|
};
|
|
|
|
|
|
2023-06-08 16:38:38 +00:00
|
|
|
/* argument for IORING_(UN)REGISTER_NAPI */
|
|
|
|
|
struct io_uring_napi {
|
|
|
|
|
__u32 busy_poll_to;
|
|
|
|
|
__u8 prefer_busy_poll;
|
2024-10-13 18:29:24 +00:00
|
|
|
|
|
|
|
|
/* a io_uring_napi_op value */
|
|
|
|
|
__u8 opcode;
|
|
|
|
|
__u8 pad[2];
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* for IO_URING_NAPI_REGISTER_OP, it is a
|
|
|
|
|
* io_uring_napi_tracking_strategy value.
|
|
|
|
|
*
|
|
|
|
|
* for IO_URING_NAPI_STATIC_ADD_ID/IO_URING_NAPI_STATIC_DEL_ID
|
|
|
|
|
* it is the napi id to add/del from napi_list.
|
|
|
|
|
*/
|
|
|
|
|
__u32 op_param;
|
|
|
|
|
__u32 resv;
|
2023-06-08 16:38:38 +00:00
|
|
|
};
|
|
|
|
|
|
2020-08-27 14:58:30 +00:00
|
|
|
/*
|
|
|
|
|
* io_uring_restriction->opcode values
|
|
|
|
|
*/
|
2024-03-28 21:09:35 +00:00
|
|
|
enum io_uring_register_restriction_op {
|
2020-08-27 14:58:30 +00:00
|
|
|
/* Allow an io_uring_register(2) opcode */
|
|
|
|
|
IORING_RESTRICTION_REGISTER_OP = 0,
|
|
|
|
|
|
|
|
|
|
/* Allow an sqe opcode */
|
|
|
|
|
IORING_RESTRICTION_SQE_OP = 1,
|
|
|
|
|
|
|
|
|
|
/* Allow sqe flags */
|
|
|
|
|
IORING_RESTRICTION_SQE_FLAGS_ALLOWED = 2,
|
|
|
|
|
|
|
|
|
|
/* Require sqe flags (these flags must be set on each submission) */
|
|
|
|
|
IORING_RESTRICTION_SQE_FLAGS_REQUIRED = 3,
|
|
|
|
|
|
|
|
|
|
IORING_RESTRICTION_LAST
|
|
|
|
|
};
|
|
|
|
|
|
io_uring: add support for fixed wait regions
Generally applications have 1 or a few waits of waiting, yet they pass
in a struct io_uring_getevents_arg every time. This needs to get copied
and, in turn, the timeout value needs to get copied.
Rather than do this for every invocation, allow the application to
register a fixed set of wait regions that can simply be indexed when
asking the kernel to wait on events.
At ring setup time, the application can register a number of these wait
regions and initialize region/index 0 upfront:
struct io_uring_reg_wait *reg;
reg = io_uring_setup_reg_wait(ring, nr_regions, &ret);
/* set timeout and mark as set, sigmask/sigmask_sz as needed */
reg->ts.tv_sec = 0;
reg->ts.tv_nsec = 100000;
reg->flags = IORING_REG_WAIT_TS;
where nr_regions >= 1 && nr_regions <= PAGE_SIZE / sizeof(*reg). The
above initializes index 0, but 63 other regions can be initialized,
if needed. Now, instead of doing:
struct __kernel_timespec timeout = { .tv_nsec = 100000, };
io_uring_submit_and_wait_timeout(ring, &cqe, nr, &t, NULL);
to wait for events for each submit_and_wait, or just wait, operation, it
can just reference the above region at offset 0 and do:
io_uring_submit_and_wait_reg(ring, &cqe, nr, 0);
to achieve the same goal of waiting 100usec without needing to copy
both struct io_uring_getevents_arg (24b) and struct __kernel_timeout
(16b) for each invocation. Struct io_uring_reg_wait looks as follows:
struct io_uring_reg_wait {
struct __kernel_timespec ts;
__u32 min_wait_usec;
__u32 flags;
__u64 sigmask;
__u32 sigmask_sz;
__u32 pad[3];
__u64 pad2[2];
};
embedding the timeout itself in the region, rather than passing it as
a pointer as well. Note that the signal mask is still passed as a
pointer, both for compatability reasons, but also because there doesn't
seem to be a lot of high frequency waits scenarios that involve setting
and resetting the signal mask for each wait.
The application is free to modify any region before a wait call, or it
can use keep multiple regions with different settings to avoid needing to
modify the same one for wait calls. Up to a page size of regions is mapped
by default, allowing PAGE_SIZE / 64 available regions for use.
The registered region must fit within a page. On a 4kb page size system,
that allows for 64 wait regions if a full page is used, as the size of
struct io_uring_reg_wait is 64b. The region registered must be aligned
to io_uring_reg_wait in size. It's valid to register less than 64
entries.
In network performance testing with zero-copy, this reduced the time
spent waiting on the TX side from 3.12% to 0.3% and the RX side from 4.4%
to 0.3%.
Wait regions are fixed for the lifetime of the ring - once registered,
they are persistent until the ring is torn down. The regions support
minimum wait timeout as well as the regular waits.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2024-10-22 19:47:00 +00:00
|
|
|
enum {
|
|
|
|
|
IORING_REG_WAIT_TS = (1U << 0),
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Argument for io_uring_enter(2) with
|
|
|
|
|
* IORING_GETEVENTS | IORING_ENTER_EXT_ARG_REG set, where the actual argument
|
|
|
|
|
* is an index into a previously registered fixed wait region described by
|
|
|
|
|
* the below structure.
|
|
|
|
|
*/
|
|
|
|
|
struct io_uring_reg_wait {
|
|
|
|
|
struct __kernel_timespec ts;
|
|
|
|
|
__u32 min_wait_usec;
|
|
|
|
|
__u32 flags;
|
|
|
|
|
__u64 sigmask;
|
|
|
|
|
__u32 sigmask_sz;
|
|
|
|
|
__u32 pad[3];
|
|
|
|
|
__u64 pad2[2];
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Argument for io_uring_enter(2) with IORING_GETEVENTS | IORING_ENTER_EXT_ARG
|
|
|
|
|
*/
|
2020-11-03 02:54:37 +00:00
|
|
|
struct io_uring_getevents_arg {
|
|
|
|
|
__u64 sigmask;
|
|
|
|
|
__u32 sigmask_sz;
|
2024-01-04 17:46:30 +00:00
|
|
|
__u32 min_wait_usec;
|
2020-11-03 02:54:37 +00:00
|
|
|
__u64 ts;
|
|
|
|
|
};
|
|
|
|
|
|
io_uring: add sync cancelation API through io_uring_register()
The io_uring cancelation API is async, like any other API that we expose
there. For the case of finding a request to cancel, or not finding one,
it is fully sync in that when submission returns, the CQE for both the
cancelation request and the targeted request have been posted to the
CQ ring.
However, if the targeted work is being executed by io-wq, the API can
only start the act of canceling it. This makes it difficult to use in
some circumstances, as the caller then has to wait for the CQEs to come
in and match on the same cancelation data there.
Provide a IORING_REGISTER_SYNC_CANCEL command for io_uring_register()
that does sync cancelations, always. For the io-wq case, it'll wait
for the cancelation to come in before returning. The only expected
returns from this API is:
0 Request found and canceled fine.
> 0 Requests found and canceled. Only happens if asked to
cancel multiple requests, and if the work wasn't in
progress.
-ENOENT Request not found.
-ETIME A timeout on the operation was requested, but the timeout
expired before we could cancel.
and we won't get -EALREADY via this API.
If the timeout value passed in is -1 (tv_sec and tv_nsec), then that
means that no timeout is requested. Otherwise, the timespec passed in
is the amount of time the sync cancel will wait for a successful
cancelation.
Link: https://github.com/axboe/liburing/discussions/608
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-18 16:00:50 +00:00
|
|
|
/*
|
|
|
|
|
* Argument for IORING_REGISTER_SYNC_CANCEL
|
|
|
|
|
*/
|
|
|
|
|
struct io_uring_sync_cancel_reg {
|
|
|
|
|
__u64 addr;
|
|
|
|
|
__s32 fd;
|
|
|
|
|
__u32 flags;
|
|
|
|
|
struct __kernel_timespec timeout;
|
2023-06-22 19:03:52 +00:00
|
|
|
__u8 opcode;
|
|
|
|
|
__u8 pad[7];
|
|
|
|
|
__u64 pad2[3];
|
io_uring: add sync cancelation API through io_uring_register()
The io_uring cancelation API is async, like any other API that we expose
there. For the case of finding a request to cancel, or not finding one,
it is fully sync in that when submission returns, the CQE for both the
cancelation request and the targeted request have been posted to the
CQ ring.
However, if the targeted work is being executed by io-wq, the API can
only start the act of canceling it. This makes it difficult to use in
some circumstances, as the caller then has to wait for the CQEs to come
in and match on the same cancelation data there.
Provide a IORING_REGISTER_SYNC_CANCEL command for io_uring_register()
that does sync cancelations, always. For the io-wq case, it'll wait
for the cancelation to come in before returning. The only expected
returns from this API is:
0 Request found and canceled fine.
> 0 Requests found and canceled. Only happens if asked to
cancel multiple requests, and if the work wasn't in
progress.
-ENOENT Request not found.
-ETIME A timeout on the operation was requested, but the timeout
expired before we could cancel.
and we won't get -EALREADY via this API.
If the timeout value passed in is -1 (tv_sec and tv_nsec), then that
means that no timeout is requested. Otherwise, the timespec passed in
is the amount of time the sync cancel will wait for a successful
cancelation.
Link: https://github.com/axboe/liburing/discussions/608
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-06-18 16:00:50 +00:00
|
|
|
};
|
|
|
|
|
|
2022-06-25 10:55:38 +00:00
|
|
|
/*
|
|
|
|
|
* Argument for IORING_REGISTER_FILE_ALLOC_RANGE
|
|
|
|
|
* The range is specified as [off, off + len)
|
|
|
|
|
*/
|
|
|
|
|
struct io_uring_file_index_range {
|
|
|
|
|
__u32 off;
|
|
|
|
|
__u32 len;
|
|
|
|
|
__u64 resv;
|
|
|
|
|
};
|
|
|
|
|
|
2022-07-14 11:02:58 +00:00
|
|
|
struct io_uring_recvmsg_out {
|
|
|
|
|
__u32 namelen;
|
|
|
|
|
__u32 controllen;
|
|
|
|
|
__u32 payloadlen;
|
|
|
|
|
__u32 flags;
|
|
|
|
|
};
|
|
|
|
|
|
2023-06-27 13:44:24 +00:00
|
|
|
/*
|
|
|
|
|
* Argument for IORING_OP_URING_CMD when file is a socket
|
|
|
|
|
*/
|
2024-03-28 21:09:35 +00:00
|
|
|
enum io_uring_socket_op {
|
2023-06-27 13:44:24 +00:00
|
|
|
SOCKET_URING_OP_SIOCINQ = 0,
|
|
|
|
|
SOCKET_URING_OP_SIOCOUTQ,
|
2023-10-16 13:47:47 +00:00
|
|
|
SOCKET_URING_OP_GETSOCKOPT,
|
2023-10-16 13:47:48 +00:00
|
|
|
SOCKET_URING_OP_SETSOCKOPT,
|
2023-06-27 13:44:24 +00:00
|
|
|
};
|
|
|
|
|
|
2025-02-15 00:09:36 +00:00
|
|
|
/* Zero copy receive refill queue entry */
|
|
|
|
|
struct io_uring_zcrx_rqe {
|
|
|
|
|
__u64 off;
|
|
|
|
|
__u32 len;
|
|
|
|
|
__u32 __pad;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct io_uring_zcrx_cqe {
|
|
|
|
|
__u64 off;
|
|
|
|
|
__u64 __pad;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/* The bit from which area id is encoded into offsets */
|
|
|
|
|
#define IORING_ZCRX_AREA_SHIFT 48
|
|
|
|
|
#define IORING_ZCRX_AREA_MASK (~(((__u64)1 << IORING_ZCRX_AREA_SHIFT) - 1))
|
|
|
|
|
|
|
|
|
|
struct io_uring_zcrx_offsets {
|
|
|
|
|
__u32 head;
|
|
|
|
|
__u32 tail;
|
|
|
|
|
__u32 rqes;
|
|
|
|
|
__u32 __resv2;
|
|
|
|
|
__u64 __resv[2];
|
|
|
|
|
};
|
|
|
|
|
|
2025-05-01 12:17:18 +00:00
|
|
|
enum io_uring_zcrx_area_flags {
|
|
|
|
|
IORING_ZCRX_AREA_DMABUF = 1,
|
|
|
|
|
};
|
|
|
|
|
|
2025-02-15 00:09:37 +00:00
|
|
|
struct io_uring_zcrx_area_reg {
|
|
|
|
|
__u64 addr;
|
|
|
|
|
__u64 len;
|
|
|
|
|
__u64 rq_area_token;
|
|
|
|
|
__u32 flags;
|
2025-05-01 12:17:18 +00:00
|
|
|
__u32 dmabuf_fd;
|
2025-02-15 00:09:37 +00:00
|
|
|
__u64 __resv2[2];
|
|
|
|
|
};
|
|
|
|
|
|
2025-02-15 00:09:36 +00:00
|
|
|
/*
|
|
|
|
|
* Argument for IORING_REGISTER_ZCRX_IFQ
|
|
|
|
|
*/
|
|
|
|
|
struct io_uring_zcrx_ifq_reg {
|
|
|
|
|
__u32 if_idx;
|
|
|
|
|
__u32 if_rxq;
|
|
|
|
|
__u32 rq_entries;
|
|
|
|
|
__u32 flags;
|
|
|
|
|
|
|
|
|
|
__u64 area_ptr; /* pointer to struct io_uring_zcrx_area_reg */
|
|
|
|
|
__u64 region_ptr; /* struct io_uring_region_desc * */
|
|
|
|
|
|
|
|
|
|
struct io_uring_zcrx_offsets offsets;
|
2025-04-15 13:09:45 +00:00
|
|
|
__u32 zcrx_id;
|
|
|
|
|
__u32 __resv2;
|
|
|
|
|
__u64 __resv[3];
|
2025-02-15 00:09:36 +00:00
|
|
|
};
|
|
|
|
|
|
2022-08-23 11:45:49 +00:00
|
|
|
#ifdef __cplusplus
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
Add io_uring IO interface
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.
With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Reviewed-by: Hannes Reinecke <hare@suse.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-01-07 17:46:33 +00:00
|
|
|
#endif
|
