JIRA: https://issues.redhat.com/browse/RHEL-118213
commit d14de5b92578c769e12d84c9bdeee5627c042270
Author: Li Zhe <lizhe.67@bytedance.com>
Date: Thu Aug 14 14:47:14 2025 +0800
vfio/type1: optimize vfio_unpin_pages_remote()
When vfio_unpin_pages_remote() is called with a range of addresses that
includes large folios, the function currently performs individual
put_pfn() operations for each page. This can lead to significant
performance overheads, especially when dealing with large ranges of pages.
It would be very rare for reserved PFNs and non reserved will to be mixed
within the same range. So this patch utilizes the has_rsvd variable
introduced in the previous patch to determine whether batch put_pfn()
operations can be performed. Moreover, compared to put_pfn(),
unpin_user_page_range_dirty_lock() is capable of handling large folio
scenarios more efficiently.
The performance test results for completing the 16G VFIO IOMMU DMA
unmapping are as follows.
Base(v6.16):
------- AVERAGE (MADV_HUGEPAGE) --------
VFIO UNMAP DMA in 0.141 s (113.7 GB/s)
------- AVERAGE (MAP_POPULATE) --------
VFIO UNMAP DMA in 0.307 s (52.2 GB/s)
------- AVERAGE (HUGETLBFS) --------
VFIO UNMAP DMA in 0.135 s (118.6 GB/s)
With this patchset:
------- AVERAGE (MADV_HUGEPAGE) --------
VFIO UNMAP DMA in 0.044 s (363.2 GB/s)
------- AVERAGE (MAP_POPULATE) --------
VFIO UNMAP DMA in 0.289 s (55.3 GB/s)
------- AVERAGE (HUGETLBFS) --------
VFIO UNMAP DMA in 0.044 s (361.3 GB/s)
For large folio, we achieve an over 67% performance improvement in
the VFIO UNMAP DMA item. For small folios, the performance test
results appear to show a slight improvement.
Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Acked-by: David Hildenbrand <david@redhat.com>
Link: https://lore.kernel.org/r/20250814064714.56485-6-lizhe.67@bytedance.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit f6c84a52cc41e2aaed0d956d0a1c1802513a239c
Author: Li Zhe <lizhe.67@bytedance.com>
Date: Thu Aug 14 14:47:11 2025 +0800
vfio/type1: optimize vfio_pin_pages_remote()
When vfio_pin_pages_remote() is called with a range of addresses that
includes large folios, the function currently performs individual
statistics counting operations for each page. This can lead to significant
performance overheads, especially when dealing with large ranges of pages.
Batch processing of statistical counting operations can effectively enhance
performance.
In addition, the pages obtained through longterm GUP are neither invalid
nor reserved. Therefore, we can reduce the overhead associated with some
calls to function is_invalid_reserved_pfn().
The performance test results for completing the 16G VFIO IOMMU DMA mapping
are as follows.
Base(v6.16):
------- AVERAGE (MADV_HUGEPAGE) --------
VFIO MAP DMA in 0.049 s (328.5 GB/s)
------- AVERAGE (MAP_POPULATE) --------
VFIO MAP DMA in 0.268 s (59.6 GB/s)
------- AVERAGE (HUGETLBFS) --------
VFIO MAP DMA in 0.051 s (310.9 GB/s)
With this patch:
------- AVERAGE (MADV_HUGEPAGE) --------
VFIO MAP DMA in 0.025 s (629.8 GB/s)
------- AVERAGE (MAP_POPULATE) --------
VFIO MAP DMA in 0.253 s (63.1 GB/s)
------- AVERAGE (HUGETLBFS) --------
VFIO MAP DMA in 0.030 s (530.5 GB/s)
For large folio, we achieve an over 40% performance improvement.
For small folios, the performance test results indicate a
slight improvement.
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Co-developed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: David Hildenbrand <david@redhat.com>
Tested-by: Eric Farman <farman@linux.ibm.com>
Link: https://lore.kernel.org/r/20250814064714.56485-3-lizhe.67@bytedance.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
Conflicts: 56d578c1300f ("mm: convert page_to_section() to memdesc_section()")
is not yet backported, therefore using page_to_section()
commit 929bf010e0599ddef6b640cd314f1de65dd1ca3e
Author: Li Zhe <lizhe.67@bytedance.com>
Date: Thu Aug 14 14:47:10 2025 +0800
mm: introduce num_pages_contiguous()
Let's add a simple helper for determining the number of contiguous pages
that represent contiguous PFNs.
In an ideal world, this helper would be simpler or not even required.
Unfortunately, on some configs we still have to maintain (SPARSEMEM
without VMEMMAP), the memmap is allocated per memory section, and we might
run into weird corner cases of false positives when blindly testing for
contiguous pages only.
One example of such false positives would be a memory section-sized hole
that does not have a memmap. The surrounding memory sections might get
"struct pages" that are contiguous, but the PFNs are actually not.
This helper will, for example, be useful for determining contiguous PFNs
in a GUP result, to batch further operations across returned "struct
page"s. VFIO will utilize this interface to accelerate the VFIO DMA map
process.
Implementation based on Linus' suggestions to avoid new usage of
nth_page() where avoidable.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Co-developed-by: David Hildenbrand <david@redhat.com>
Signed-off-by: David Hildenbrand <david@redhat.com>
Link: https://lore.kernel.org/r/20250814064714.56485-2-lizhe.67@bytedance.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 8b9f128947dd72e0fcf256088a673abac9b720bf
Author: Timothy Pearson <tpearson@raptorengineering.com>
Date: Tue Sep 23 12:04:33 2025 -0500
vfio/pci: Fix INTx handling on legacy non-PCI 2.3 devices
PCI devices prior to PCI 2.3 both use level interrupts and do not support
interrupt masking, leading to a failure when passed through to a KVM guest on
at least the ppc64 platform. This failure manifests as receiving and
acknowledging a single interrupt in the guest, while the device continues to
assert the level interrupt indicating a need for further servicing.
When lazy IRQ masking is used on DisINTx- (non-PCI 2.3) hardware, the following
sequence occurs:
* Level IRQ assertion on device
* IRQ marked disabled in kernel
* Host interrupt handler exits without clearing the interrupt on the device
* Eventfd is delivered to userspace
* Guest processes IRQ and clears device interrupt
* Device de-asserts INTx, then re-asserts INTx while the interrupt is masked
* Newly asserted interrupt acknowledged by kernel VMM without being handled
* Software mask removed by VFIO driver
* Device INTx still asserted, host controller does not see new edge after EOI
The behavior is now platform-dependent. Some platforms (amd64) will continue
to spew IRQs for as long as the INTX line remains asserted, therefore the IRQ
will be handled by the host as soon as the mask is dropped. Others (ppc64) will
only send the one request, and if it is not handled no further interrupts will
be sent. The former behavior theoretically leaves the system vulnerable to
interrupt storm, and the latter will result in the device stalling after
receiving exactly one interrupt in the guest.
Work around this by disabling lazy IRQ masking for DisINTx- INTx devices.
Signed-off-by: Timothy Pearson <tpearson@raptorengineering.com>
Link: https://lore.kernel.org/r/333803015.1744464.1758647073336.JavaMail.zimbra@raptorengineeringinc.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 16df67f2189a71a8310bcebddb87ed569e8352be
Author: Alex Mastro <amastro@fb.com>
Date: Mon Sep 8 08:58:40 2025 -0700
vfio: return -ENOTTY for unsupported device feature
The two implementers of vfio_device_ops.device_feature,
vfio_cdx_ioctl_feature and vfio_pci_core_ioctl_feature, return
-ENOTTY in the fallthrough case when the feature is unsupported. For
consistency, the base case, vfio_ioctl_device_feature, should do the
same when device_feature == NULL, indicating an implementation has no
feature extensions.
Signed-off-by: Alex Mastro <amastro@fb.com>
Link: https://lore.kernel.org/r/20250908-vfio-enotty-v1-1-4428e1539e2e@fb.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 03e073bc4dbc3d64ce0beb21fbe793ae7787e062
Author: David Matlack <dmatlack@google.com>
Date: Thu Aug 28 18:58:14 2025 +0000
vfio: selftests: Fix .gitignore for already tracked files
Fix the rules in tools/testing/selftests/vfio/.gitignore to not ignore
some already tracked files (.gitignore, Makefile, lib/libvfio.mk).
This change should be a no-op, since these files are already tracked by git
and thus git will not ignore updates to them even though they match the
ignore rules in the VFIO selftests .gitignore file.
However, they do generate warnings with W=1, as reported by the kernel test
robot.
$ KBUILD_EXTRA_WARN=1 scripts/misc-check
tools/testing/selftests/vfio/.gitignore: warning: ignored by one of the .gitignore files
tools/testing/selftests/vfio/Makefile: warning: ignored by one of the .gitignore files
tools/testing/selftests/vfio/lib/libvfio.mk: warning: ignored by one of the .gitignore files
Fix this by explicitly un-ignoring the tracked files.
Fixes: 292e9ee22b0a ("selftests: Create tools/testing/selftests/vfio")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202508280918.rFRyiLEU-lkp@intel.com/
Signed-off-by: David Matlack <dmatlack@google.com>
Link: https://lore.kernel.org/r/20250828185815.382215-1-dmatlack@google.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit fd134b0f2f8ef9b1b7b0cade8cac4ff831619713
Author: David Matlack <dmatlack@google.com>
Date: Fri Aug 22 21:25:17 2025 +0000
vfio: selftests: Add a script to help with running VFIO selftests
Introduce run.sh, a script to help with running VFIO selftests. The
script is intended to be used for both humans manually running VFIO
selftests, and to incorporate into test automation where VFIO selftests
may run alongside other tests. As such the script aims to be hermetic,
returning the system to the state it was before the test started.
The script takes as input the BDF of a device to use and a command to
run (typically the command would be a VFIO selftest). e.g.
$ ./run.sh -d 0000:6a:01.0 ./vfio_pci_device_test
or
$ ./run.sh -d 0000:6a:01.0 -- ./vfio_pci_device_test
The script then handles unbinding device 0000:6a:01.0 from its current
driver, binding it to vfio-pci, running the test, unbinding from
vfio-pci, and binding back to the original driver.
When run.sh runs the provided test, it does so by appending the BDF as
the last parameter. For example:
$ ./run.sh -d 0000:6a:01.0 -- echo hello
Results in the following being printed to stdout:
hello 0000:6a:01.0
The script also supports a mode where it can break out into a shell so
that multiple tests can be run manually.
$ ./run.sh -d 0000:6a:01.0 -s
$ echo $VFIO_SELFTESTS_BDF
$ ./vfio_pci_device_test
$ exit
Choosing which device to use is up to the user.
In the future this script should be extensible to tests that want to use
multiple devices. The script can support accepting -d BDF multiple times
and parse them into an array, setup all the devices, pass the list of
BDFs to the test, and then cleanup all the devices.
Acked-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Matlack <dmatlack@google.com>
Link: https://lore.kernel.org/r/20250822212518.4156428-31-dmatlack@google.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 2223587df5c5e935cc6f973f62a9608eef81bec8
Author: David Matlack <dmatlack@google.com>
Date: Fri Aug 22 21:25:06 2025 +0000
vfio: selftests: Add driver for Intel CBDMA
Add a driver for the Intel CBDMA device. This driver is based on and
named after the Linux driver for this device (drivers/dma/ioat/) and
also based on previous work from Peter Shier <pshier@google.com>.
The driver aims to be as simple as possible. It uses a single descriptor
to issue DMA operations, and only supports the copy operation. For "DMA
storms", the driver kicks off the maximum number of maximum-sized DMA
operations. On Skylake server parts, this was 2^16-1 copies of size 2M
and lasts about 15 seconds.
Create symlinks to drivers/dma/ioat/{hw.h,registers.h} to get access to
various macros (e.g. IOAT_CHANCMD_RESET) and struct ioat_dma_descriptor.
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Acked-by: Dave Jiang <dave.jiang@intel.com>
Acked-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Matlack <dmatlack@google.com>
Link: https://lore.kernel.org/r/20250822212518.4156428-20-dmatlack@google.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 1f9c8edd6a7e9b0fd914cfeef8ce075307e8e702
Author: David Matlack <dmatlack@google.com>
Date: Fri Aug 22 21:25:02 2025 +0000
tools headers: Import asm-generic MMIO helpers
Import the asm-generic MMIO helper functions from the kernel headers
into tools/include/. The top-level include is <linux/io.h> which then
includes the arch-specific <asm/io.h>, which then includes
<asm-generic/io.h>. This layout is chosen to match the kernel header
layout and to appease checkpatch.pl (which warns against including
<asm/io.h> or <asm-generic/io.h> directly).
Changes made when importing:
- Add missing includes at the top.
- Stub out mmiowb_set_pending().
- Stub out _THIS_IP_.
- Stub out log_*_mmio() calls.
- Drop the CONFIG_64BIT checks, since tools/include/linux/types.h
always defines u64.
Acked-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Matlack <dmatlack@google.com>
Link: https://lore.kernel.org/r/20250822212518.4156428-16-dmatlack@google.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 16eadd7c1277284cfff5b7071910920a2d008251
Author: David Matlack <dmatlack@google.com>
Date: Fri Aug 22 21:24:50 2025 +0000
vfio: selftests: Introduce vfio_pci_device_test
Introduce a basic VFIO selftest called vfio_pci_device_test to
demonstrate the functionality of the VFIO selftest library and provide
some test coverage of basic VFIO operations, including:
- Mapping and unmapping DMA
- Mapping and unmapping BARs
- Enabling, triggering, and disabling MSI and MSI-x
- Reading and writing to PCI config space
This test should work with most PCI devices, as long as they are bound
to vfio-pci.
Acked-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Matlack <dmatlack@google.com>
Link: https://lore.kernel.org/r/20250822212518.4156428-4-dmatlack@google.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 19faf6fd969c21589b6dd40c35255e4d00d427f8
Author: David Matlack <dmatlack@google.com>
Date: Fri Aug 22 21:24:49 2025 +0000
vfio: selftests: Add a helper library for VFIO selftests
Add a basic helper library to be used by VFIO selftests.
The basic unit of the library is struct vfio_pci_device, which
represents a single PCI device that is bound to the vfio-pci driver. The
library currently only supports a single device per group and container,
and VFIO IOMMU types.
The code in this library was heavily based on prior work done by
Raghavendra Rao Ananta <rananta@google.com>, and the VFIO_ASSERT*()
macros were written by Vipin Sharma <vipinsh@google.com>.
Separate that Makefile rules for building the library into a separate
script so that the library can be built by and linked into KVM selftests
in a subsequent commit.
Acked-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Matlack <dmatlack@google.com>
Link: https://lore.kernel.org/r/20250822212518.4156428-3-dmatlack@google.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit b1779e4f209c7ff7e32f3c79d69bca4e3a3a68b6
Author: Keith Busch <kbusch@kernel.org>
Date: Tue Jul 15 11:46:22 2025 -0700
vfio/type1: conditional rescheduling while pinning
A large DMA mapping request can loop through dma address pinning for
many pages. In cases where THP can not be used, the repeated vmf_insert_pfn can
be costly, so let the task reschedule as need to prevent CPU stalls. Failure to
do so has potential harmful side effects, like increased memory pressure
as unrelated rcu tasks are unable to make their reclaim callbacks and
result in OOM conditions.
rcu: INFO: rcu_sched self-detected stall on CPU
rcu: 36-....: (20999 ticks this GP) idle=b01c/1/0x4000000000000000 softirq=35839/35839 fqs=3538
rcu: hardirqs softirqs csw/system
rcu: number: 0 107 0
rcu: cputime: 50 0 10446 ==> 10556(ms)
rcu: (t=21075 jiffies g=377761 q=204059 ncpus=384)
...
<TASK>
? asm_sysvec_apic_timer_interrupt+0x16/0x20
? walk_system_ram_range+0x63/0x120
? walk_system_ram_range+0x46/0x120
? pgprot_writethrough+0x20/0x20
lookup_memtype+0x67/0xf0
track_pfn_insert+0x20/0x40
vmf_insert_pfn_prot+0x88/0x140
vfio_pci_mmap_huge_fault+0xf9/0x1b0 [vfio_pci_core]
__do_fault+0x28/0x1b0
handle_mm_fault+0xef1/0x2560
fixup_user_fault+0xf5/0x270
vaddr_get_pfns+0x169/0x2f0 [vfio_iommu_type1]
vfio_pin_pages_remote+0x162/0x8e0 [vfio_iommu_type1]
vfio_iommu_type1_ioctl+0x1121/0x1810 [vfio_iommu_type1]
? futex_wake+0x1c1/0x260
x64_sys_call+0x234/0x17a0
do_syscall_64+0x63/0x130
? exc_page_fault+0x63/0x130
entry_SYSCALL_64_after_hwframe+0x4b/0x53
Signed-off-by: Keith Busch <kbusch@kernel.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://lore.kernel.org/r/20250715184622.3561598-1-kbusch@meta.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 86624ba3b522b6512def25534341da93356c8da4
Author: Jason Gunthorpe <jgg@ziepe.ca>
Date: Mon Jul 14 13:08:25 2025 -0300
vfio/pci: Do vf_token checks for VFIO_DEVICE_BIND_IOMMUFD
This was missed during the initial implementation. The VFIO PCI encodes
the vf_token inside the device name when opening the device from the group
FD, something like:
"0000:04:10.0 vf_token=bd8d9d2b-5a5f-4f5a-a211-f591514ba1f3"
This is used to control access to a VF unless there is co-ordination with
the owner of the PF.
Since we no longer have a device name in the cdev path, pass the token
directly through VFIO_DEVICE_BIND_IOMMUFD using an optional field
indicated by VFIO_DEVICE_BIND_FLAG_TOKEN.
Fixes: 5fcc26969a ("vfio: Add VFIO_DEVICE_BIND_IOMMUFD")
Tested-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Reviewed-by: Yi Liu <yi.l.liu@intel.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Link: https://lore.kernel.org/r/0-v3-bdd8716e85fe+3978a-vfio_token_jgg@nvidia.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit e908f58b6beb337cbe4481d52c3f5c78167b1aab
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Thu Jun 26 16:56:18 2025 -0600
vfio/pci: Separate SR-IOV VF dev_set
In the below noted Fixes commit we introduced a reflck mutex to allow
better scaling between devices for open and close. The reflck was
based on the hot reset granularity, device level for root bus devices
which cannot support hot reset or bus/slot reset otherwise. Overlooked
in this were SR-IOV VFs, where there's also no bus reset option, but
the default for a non-root-bus, non-slot-based device is bus level
reflck granularity.
The reflck mutex has since become the dev_set mutex (via commit
2cd8b14aaa ("vfio/pci: Move to the device set infrastructure")) and
is our defacto serialization for various operations and ioctls. It
still seems to be the case though that sets of vfio-pci devices really
only need serialization relative to hot resets affecting the entire
set, which is not relevant to SR-IOV VFs. As described in the Closes
link below, this serialization contributes to startup latency when
multiple VFs sharing the same "bus" are opened concurrently.
Mark the device itself as the basis of the dev_set for SR-IOV VFs.
Reported-by: Aaron Lewis <aaronlewis@google.com>
Closes: https://lore.kernel.org/all/20250626180424.632628-1-aaronlewis@google.com
Tested-by: Aaron Lewis <aaronlewis@google.com>
Fixes: e309df5b0c ("vfio/pci: Parallelize device open and release")
Reviewed-by: Yi Liu <yi.l.liu@intel.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Link: https://lore.kernel.org/r/20250626225623.1180952-1-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
CVE: CVE-2025-38625
commit fe24d5bc635e103a517ec201c3cb571eeab8be2f
Author: Brett Creeley <brett.creeley@amd.com>
Date: Wed Jul 2 09:37:44 2025 -0700
vfio/pds: Fix missing detach_ioas op
When CONFIG_IOMMUFD is enabled and a device is bound to the pds_vfio_pci
driver, the following WARN_ON() trace is seen and probe fails:
WARNING: CPU: 0 PID: 5040 at drivers/vfio/vfio_main.c:317 __vfio_register_dev+0x130/0x140 [vfio]
<...>
pds_vfio_pci 0000:08:00.1: probe with driver pds_vfio_pci failed with error -22
This is because the driver's vfio_device_ops.detach_ioas isn't set.
Fix this by using the generic vfio_iommufd_physical_detach_ioas
function.
Fixes: 38fe3975b4 ("vfio/pds: Initial support for pds VFIO driver")
Signed-off-by: Brett Creeley <brett.creeley@amd.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Link: https://lore.kernel.org/r/20250702163744.69767-1-brett.creeley@amd.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit b25e271b377999191b12f0afbe1861edcf57e3fe
Author: Jacob Pan <jacob.pan@linux.microsoft.com>
Date: Wed Jun 18 16:46:17 2025 -0700
vfio: Fix unbalanced vfio_df_close call in no-iommu mode
For devices with no-iommu enabled in IOMMUFD VFIO compat mode, the group open
path skips vfio_df_open(), leaving open_count at 0. This causes a warning in
vfio_assert_device_open(device) when vfio_df_close() is called during group
close.
The correct behavior is to skip only the IOMMUFD bind in the device open path
for no-iommu devices. Commit 6086efe734 omitted vfio_df_open(), which was
too broad. This patch restores the previous behavior, ensuring
the vfio_df_open is called in the group open path.
Fixes: 6086efe734 ("vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()")
Suggested-by: Alex Williamson <alex.williamson@redhat.com>
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Jacob Pan <jacob.pan@linux.microsoft.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Link: https://lore.kernel.org/r/20250618234618.1910456-1-jacob.pan@linux.microsoft.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 2b521d86ee80a436a92445b8206d38d75aeb39ea
Author: Sean Christopherson <seanjc@google.com>
Date: Fri May 16 16:07:29 2025 -0700
irqbypass: Take ownership of producer/consumer token tracking
Move ownership of IRQ bypass token tracking into irqbypass.ko, and
explicitly require callers to pass an eventfd_ctx structure instead of a
completely opaque token. Relying on producers and consumers to set the
token appropriately is error prone, and hiding the fact that the token must
be an eventfd_ctx pointer (for all intents and purposes) unnecessarily
obfuscates the code and makes it more brittle.
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Link: https://lore.kernel.org/r/20250516230734.2564775-4-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit fa079a0616edbcdad538128306abbc19b68a9863
Author: Sean Christopherson <seanjc@google.com>
Date: Fri May 16 16:07:27 2025 -0700
irqbypass: Drop pointless and misleading THIS_MODULE get/put
Drop irqbypass.ko's superfluous and misleading get/put calls on
THIS_MODULE. A module taking a reference to itself is useless; no amount
of checks will prevent doom and destruction if the caller hasn't already
guaranteed the liveliness of the module (this goes for any module). E.g.
if try_module_get() fails because irqbypass.ko is being unloaded, then the
kernel has already hit a use-after-free by virtue of executing code whose
lifecycle is tied to irqbypass.ko.
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Link: https://lore.kernel.org/r/20250516230734.2564775-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 4518e5a60c7fbf0cdff393c2681db39d77b4f87e
Author: Li RongQing <lirongqing@baidu.com>
Date: Wed May 21 11:46:47 2025 +0800
vfio/type1: Fix error unwind in migration dirty bitmap allocation
When setting up dirty page tracking at the vfio IOMMU backend for
device migration, if an error is encountered allocating a tracking
bitmap, the unwind loop fails to free previously allocated tracking
bitmaps. This occurs because the wrong loop index is used to
generate the tracking object. This results in unintended memory
usage for the life of the current DMA mappings where bitmaps were
successfully allocated.
Use the correct loop index to derive the tracking object for
freeing during unwind.
Fixes: d6a4c18566 ("vfio iommu: Implementation of ioctl for dirty pages tracking")
Signed-off-by: Li RongQing <lirongqing@baidu.com>
Link: https://lore.kernel.org/r/20250521034647.2877-1-lirongqing@baidu.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit ac6c973a480bcbd1161974b6b8ec6fd12e87b506
Author: Leon Romanovsky <leon@kernel.org>
Date: Tue May 20 16:46:31 2025 +0300
vfio/mlx5: Rewrite create mkey flow to allow better code reuse
Change the creation of mkey to be performed in multiple steps:
data allocation, DMA setup and actual call to HW to create that mkey.
In this new flow, the whole input to MKEY command is saved to eliminate
the need to keep array of pointers for DMA addresses for receive list
and in the future patches for send list too.
In addition to memory size reduce and elimination of unnecessary data
movements to set MKEY input, the code is prepared for future reuse.
Tested-by: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
Acked-by: Yishai Hadas <yishaih@nvidia.com>
Link: https://lore.kernel.org/r/d4ad0384fbd1e23a607cbbe9e5756748f3a761d9.1747747694.git.leon@kernel.org
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
CVE: CVE-2025-38283
commit 2777a40998deb36f96b6afc48bd397cf58a4edf0
Author: Longfang Liu <liulongfang@huawei.com>
Date: Sat May 10 16:11:54 2025 +0800
hisi_acc_vfio_pci: bugfix live migration function without VF device driver
If the VF device driver is not loaded in the Guest OS and we attempt to
perform device data migration, the address of the migrated data will
be NULL.
The live migration recovery operation on the destination side will
access a null address value, which will cause access errors.
Therefore, live migration of VMs without added VF device drivers
does not require device data migration.
In addition, when the queue address data obtained by the destination
is empty, device queue recovery processing will not be performed.
Fixes: b0eed08590 ("hisi_acc_vfio_pci: Add support for VFIO live migration")
Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Link: https://lore.kernel.org/r/20250510081155.55840-6-liulongfang@huawei.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit db6525a8573957faea28850392f4744e5f8f7a53
Author: Longfang Liu <liulongfang@huawei.com>
Date: Sat May 10 16:11:53 2025 +0800
hisi_acc_vfio_pci: bugfix the problem of uninstalling driver
In a live migration scenario. If the number of VFs at the
destination is greater than the source, the recovery operation
will fail and qemu will not be able to complete the process and
exit after shutting down the device FD.
This will cause the driver to be unable to be unloaded normally due
to abnormal reference counting of the live migration driver caused
by the abnormal closing operation of fd.
Therefore, make sure the migration file descriptor references are
always released when the device is closed.
Fixes: b0eed08590 ("hisi_acc_vfio_pci: Add support for VFIO live migration")
Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Link: https://lore.kernel.org/r/20250510081155.55840-5-liulongfang@huawei.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 3495cec0787721ba7a9d5c19d0bbb66d182de584
Author: Longfang Liu <liulongfang@huawei.com>
Date: Sat May 10 16:11:51 2025 +0800
hisi_acc_vfio_pci: add eq and aeq interruption restore
In order to ensure that the task packets of the accelerator
device are not lost during the migration process, it is necessary
to send an EQ and AEQ command to the device after the live migration
is completed and to update the completion position of the task queue.
Let the device recheck the completed tasks data and if there are
uncollected packets, device resend a task completion interrupt
to the software.
Fixes: b0eed08590 ("hisi_acc_vfio_pci: Add support for VFIO live migration")
Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Link: https://lore.kernel.org/r/20250510081155.55840-3-liulongfang@huawei.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
CVE: CVE-2025-38158
commit 8bb7170c5a055ea17c6857c256ee73c10ff872eb
Author: Longfang Liu <liulongfang@huawei.com>
Date: Sat May 10 16:11:50 2025 +0800
hisi_acc_vfio_pci: fix XQE dma address error
The dma addresses of EQE and AEQE are wrong after migration and
results in guest kernel-mode encryption services failure.
Comparing the definition of hardware registers, we found that
there was an error when the data read from the register was
combined into an address. Therefore, the address combination
sequence needs to be corrected.
Even after fixing the above problem, we still have an issue
where the Guest from an old kernel can get migrated to
new kernel and may result in wrong data.
In order to ensure that the address is correct after migration,
if an old magic number is detected, the dma address needs to be
updated.
Fixes: b0eed08590 ("hisi_acc_vfio_pci: Add support for VFIO live migration")
Signed-off-by: Longfang Liu <liulongfang@huawei.com>
Reviewed-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>
Link: https://lore.kernel.org/r/20250510081155.55840-2-liulongfang@huawei.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-118213
commit 2bd42b03ab6b04dde1753bd6b38eeca5c70f3941
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Thu Mar 20 13:41:42 2025 -0600
vfio/pci: Virtualize zero INTx PIN if no pdev->irq
Typically pdev->irq is consistent with whether the device itself
supports INTx, where device support is reported via the PIN register.
Therefore the PIN register is often already zero if pdev->irq is zero.
Recently virtualization of the PIN register was expanded to include
the case where the device supports INTx but the platform does not
route the interrupt. This is reported by a value of IRQ_NOTCONNECTED
on some architectures. Other architectures just report zero for
pdev->irq.
We already disallow INTx setup if pdev->irq is zero, therefore add
this to the PIN register virtualization criteria so that a consistent
view is provided to userspace through virtualized config space and
ioctls.
Reported-by: Shivaprasad G Bhat <sbhat@linux.ibm.com>
Link: https://lore.kernel.org/all/174231895238.2295.12586708771396482526.stgit@linux.ibm.com/
Tested-by: Shivaprasad G Bhat <sbhat@linux.ibm.com>
Link: https://lore.kernel.org/r/20250320194145.2816379-1-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-89852
commit c1d9dac0db168198b6f63f460665256dedad9b6e
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Fri May 2 16:40:31 2025 -0600
vfio/pci: Align huge faults to order
The vfio-pci huge_fault handler doesn't make any attempt to insert a
mapping containing the faulting address, it only inserts mappings if the
faulting address and resulting pfn are aligned. This works in a lot of
cases, particularly in conjunction with QEMU where DMA mappings linearly
fault the mmap. However, there are configurations where we don't get
that linear faulting and pages are faulted on-demand.
The scenario reported in the bug below is such a case, where the physical
address width of the CPU is greater than that of the IOMMU, resulting in a
VM where guest firmware has mapped device MMIO beyond the address width of
the IOMMU. In this configuration, the MMIO is faulted on demand and
tracing indicates that occasionally the faults generate a VM_FAULT_OOM.
Given the use case, this results in a "error: kvm run failed Bad address",
killing the VM.
The host is not under memory pressure in this test, therefore it's
suspected that VM_FAULT_OOM is actually the result of a NULL return from
__pte_offset_map_lock() in the get_locked_pte() path from insert_pfn().
This suggests a potential race inserting a pte concurrent to a pmd, and
maybe indicates some deficiency in the mm layer properly handling such a
case.
Nevertheless, Peter noted the inconsistency of vfio-pci's huge_fault
handler where our mapping granularity depends on the alignment of the
faulting address relative to the order rather than aligning the faulting
address to the order to more consistently insert huge mappings. This
change not only uses the page tables more consistently and efficiently, but
as any fault to an aligned page results in the same mapping, the race
condition suspected in the VM_FAULT_OOM is avoided.
Reported-by: Adolfo <adolfotregosa@gmail.com>
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=220057
Fixes: 09dfc8a5f2ce ("vfio/pci: Fallback huge faults for unaligned pfn")
Cc: stable@vger.kernel.org
Tested-by: Adolfo <adolfotregosa@gmail.com>
Co-developed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Link: https://lore.kernel.org/r/20250502224035.3183451-1-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
JIRA: https://issues.redhat.com/browse/RHEL-85591
commit 860be250fc32de9cb24154bf21b4e36f40925707
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Tue Mar 11 17:06:21 2025 -0600
vfio/pci: Handle INTx IRQ_NOTCONNECTED
Some systems report INTx as not routed by setting pdev->irq to
IRQ_NOTCONNECTED, resulting in a -ENOTCONN error when trying to
setup eventfd signaling. Include this in the set of conditions
for which the PIN register is virtualized to zero.
Additionally consolidate vfio_pci_get_irq_count() to use this
virtualized value in reporting INTx support via ioctl and sanity
checking ioctl paths since pdev->irq is re-used when the device
is in MSI mode.
The combination of these results in both the config space of the
device and the ioctl interface behaving as if the device does not
support INTx.
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Link: https://lore.kernel.org/r/20250311230623.1264283-1-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit 384a530111c682d4a6c4c9ec305254c323a1379a
Author: Yishai Hadas <yishaih@nvidia.com>
Date: Sun Mar 2 18:27:23 2025 +0200
vfio/virtio: Enable support for virtio-block live migration
With a functional and tested backend for virtio-block live migration,
add the virtio-block device ID to the pci_device_id table.
Currently, the driver supports legacy IO functionality only for
virtio-net, and it is accounted for in specific parts of the code.
To enforce this limitation, an explicit check for virtio-net, has been
added in virtiovf_support_legacy_io(). Once a backend implements legacy
IO functionality for virtio-block, the necessary support will be added
to the driver, and this additional check should be removed.
The module description was updated accordingly.
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Yishai Hadas <yishaih@nvidia.com>
Link: https://lore.kernel.org/r/20250302162723.82578-1-yishaih@nvidia.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit 0fd06844de5d063cb384384e06a11ec7141a35d5
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Tue Feb 18 15:22:06 2025 -0700
vfio/type1: Use mapping page mask for pfnmaps
vfio-pci supports huge_fault for PCI MMIO BARs and will insert pud and
pmd mappings for well aligned mappings. follow_pfnmap_start() walks the
page table and therefore knows the page mask of the level where the
address is found and returns this through follow_pfnmap_args.addr_mask.
Subsequent pfns from this address until the end of the mapping page are
necessarily consecutive. Use this information to retrieve a range of
pfnmap pfns in a single pass.
With optimal mappings and alignment on systems with 1GB pud and 4KB
page size, this reduces iterations for DMA mapping PCI BARs by a
factor of 256K. In real world testing, the overhead of iterating
pfns for a VM DMA mapping a 32GB PCI BAR is reduced from ~1s to
sub-millisecond overhead.
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Tested-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Link: https://lore.kernel.org/r/20250218222209.1382449-7-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit 62fb8adc43afad5fa1c9cadc6f3a8e9fb72af194
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Tue Feb 18 15:22:05 2025 -0700
mm: Provide address mask in struct follow_pfnmap_args
follow_pfnmap_start() walks the page table for a given address and
fills out the struct follow_pfnmap_args in pfnmap_args_setup().
The address mask of the page table level is already provided to this
latter function for calculating the pfn. This address mask can also
be useful for the caller to determine the extent of the contiguous
mapping.
For example, vfio-pci now supports huge_fault for pfnmaps and is able
to insert pud and pmd mappings. When we DMA map these pfnmaps, ex.
PCI MMIO BARs, we iterate follow_pfnmap_start() to get each pfn to test
for a contiguous pfn range. Providing the mapping address mask allows
us to skip the extent of the mapping level. Assuming a 1GB pud level
and 4KB page size, iterations are reduced by a factor of 256K. In wall
clock time, mapping a 32GB PCI BAR is reduced from ~1s to <1ms.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@redhat.com>
Cc: linux-mm@kvack.org
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Tested-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Acked-by: David Hildenbrand <david@redhat.com>
Link: https://lore.kernel.org/r/20250218222209.1382449-6-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit 0635559233434a337aa1c20d53abae18b3663796
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Tue Feb 18 15:22:04 2025 -0700
vfio/type1: Use consistent types for page counts
Page count should more consistently be an unsigned long when passed as
an argument while functions returning a number of pages should use a
signed long to allow for -errno.
vaddr_get_pfns() can therefore be upgraded to return long, though in
practice it's currently limited by the batch capacity. In fact, the
batch indexes are noted to never hold negative values, so while it
doesn't make sense to bloat the structure with unsigned longs in this
case, it does make sense to specify these as unsigned.
No change in behavior expected.
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Tested-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Link: https://lore.kernel.org/r/20250218222209.1382449-5-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit eb996eec783c1e7e1e9c62e0336f8b86a08cf541
Author: Alex Williamson <alex.williamson@redhat.com>
Date: Tue Feb 18 15:22:03 2025 -0700
vfio/type1: Use vfio_batch for vaddr_get_pfns()
Passing the vfio_batch to vaddr_get_pfns() allows for greater
distinction between page backed pfns and pfnmaps. In the case of page
backed pfns, vfio_batch.size is set to a positive value matching the
number of pages filled in vfio_batch.pages. For a pfnmap,
vfio_batch.size remains zero as vfio_batch.pages are not used. In both
cases the return value continues to indicate the number of pfns and the
provided pfn arg is set to the initial pfn value.
This allows us to shortcut the pfnmap case, which is detected by the
zero vfio_batch.size. pfnmaps do not contribute to locked memory
accounting, therefore we can update counters and continue directly,
which also enables a future where vaddr_get_pfns() can return a value
greater than one for consecutive pfnmaps.
NB. Now that we're not guessing whether the initial pfn is page backed
or pfnmap, we no longer need to special case the put_pfn() and batch
size reset. It's safe for vfio_batch_unpin() to handle this case.
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Tested-by: Mitchell Augustin <mitchell.augustin@canonical.com>
Link: https://lore.kernel.org/r/20250218222209.1382449-4-alex.williamson@redhat.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit d85f69d520e6aca8ae5ce353666e2fc2756eb9e7
Author: Ankit Agrawal <ankita@nvidia.com>
Date: Fri Jan 24 18:31:01 2025 +0000
vfio/nvgrace-gpu: Check the HBM training and C2C link status
In contrast to Grace Hopper systems, the HBM training has been moved
out of the UEFI on the Grace Blackwell systems. This reduces the system
bootup time significantly.
The onus of checking whether the HBM training has completed thus falls
on the module.
The HBM training status can be determined from a BAR0 register.
Similarly, another BAR0 register exposes the status of the CPU-GPU
chip-to-chip (C2C) cache coherent interconnect.
Based on testing, 30s is determined to be sufficient to ensure
initialization completion on all the Grace based systems. Thus poll
these register and check for 30s. If the HBM training is not complete
or if the C2C link is not ready, fail the probe.
While the time is not required on Grace Hopper systems, it is
beneficial to make the check to ensure the device is in an
expected state. Hence keeping it generalized to both the generations.
Ensure that the BAR0 is enabled before accessing the registers.
CC: Alex Williamson <alex.williamson@redhat.com>
CC: Kevin Tian <kevin.tian@intel.com>
CC: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
Link: https://lore.kernel.org/r/20250124183102.3976-4-ankita@nvidia.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit 6a9eb2d125ba90d13b45bcfabcddf9f61268f6a8
Author: Ankit Agrawal <ankita@nvidia.com>
Date: Fri Jan 24 18:31:00 2025 +0000
vfio/nvgrace-gpu: Expose the blackwell device PF BAR1 to the VM
There is a HW defect on Grace Hopper (GH) to support the
Multi-Instance GPU (MIG) feature [1] that necessiated the presence
of a 1G region carved out from the device memory and mapped as
uncached. The 1G region is shown as a fake BAR (comprising region 2 and 3)
to workaround the issue.
The Grace Blackwell systems (GB) differ from GH systems in the following
aspects:
1. The aforementioned HW defect is fixed on GB systems.
2. There is a usable BAR1 (region 2 and 3) on GB systems for the
GPUdirect RDMA feature [2].
This patch accommodate those GB changes by showing the 64b physical
device BAR1 (region2 and 3) to the VM instead of the fake one. This
takes care of both the differences.
Moreover, the entire device memory is exposed on GB as cacheable to
the VM as there is no carveout required.
Link: https://www.nvidia.com/en-in/technologies/multi-instance-gpu/ [1]
Link: https://docs.nvidia.com/cuda/gpudirect-rdma/ [2]
Cc: Kevin Tian <kevin.tian@intel.com>
CC: Jason Gunthorpe <jgg@nvidia.com>
Suggested-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
Link: https://lore.kernel.org/r/20250124183102.3976-3-ankita@nvidia.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit bd53764a60ad586ad5b6ed339423ad5e67824464
Author: Ankit Agrawal <ankita@nvidia.com>
Date: Fri Jan 24 18:30:59 2025 +0000
vfio/nvgrace-gpu: Read dvsec register to determine need for uncached resmem
NVIDIA's recently introduced Grace Blackwell (GB) Superchip is a
continuation with the Grace Hopper (GH) superchip that provides a
cache coherent access to CPU and GPU to each other's memory with
an internal proprietary chip-to-chip cache coherent interconnect.
There is a HW defect on GH systems to support the Multi-Instance
GPU (MIG) feature [1] that necessiated the presence of a 1G region
with uncached mapping carved out from the device memory. The 1G
region is shown as a fake BAR (comprising region 2 and 3) to
workaround the issue. This is fixed on the GB systems.
The presence of the fix for the HW defect is communicated by the
device firmware through the DVSEC PCI config register with ID 3.
The module reads this to take a different codepath on GB vs GH.
Scan through the DVSEC registers to identify the correct one and use
it to determine the presence of the fix. Save the value in the device's
nvgrace_gpu_pci_core_device structure.
Link: https://www.nvidia.com/en-in/technologies/multi-instance-gpu/ [1]
CC: Jason Gunthorpe <jgg@nvidia.com>
CC: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Ankit Agrawal <ankita@nvidia.com>
Link: https://lore.kernel.org/r/20250124183102.3976-2-ankita@nvidia.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-85587
commit 2b938e3db335e3670475e31a722c2bee34748c5a
Author: Ramesh Thomas <ramesh.thomas@intel.com>
Date: Tue Dec 10 05:19:37 2024 -0800
vfio/pci: Enable iowrite64 and ioread64 for vfio pci
Definitions of ioread64 and iowrite64 macros in asm/io.h called by vfio
pci implementations are enclosed inside check for CONFIG_GENERIC_IOMAP.
They don't get defined if CONFIG_GENERIC_IOMAP is defined. Include
linux/io-64-nonatomic-lo-hi.h to define iowrite64 and ioread64 macros
when they are not defined. io-64-nonatomic-lo-hi.h maps the macros to
generic implementation in lib/iomap.c. The generic implementation does
64 bit rw if readq/writeq is defined for the architecture, otherwise it
would do 32 bit back to back rw.
Note that there are two versions of the generic implementation that
differs in the order the 32 bit words are written if 64 bit support is
not present. This is not the little/big endian ordering, which is
handled separately. This patch uses the lo followed by hi word ordering
which is consistent with current back to back implementation in the
vfio/pci code.
Signed-off-by: Ramesh Thomas <ramesh.thomas@intel.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Link: https://lore.kernel.org/r/20241210131938.303500-2-ramesh.thomas@intel.com
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Alex Williamson