qiurui
/
linux-kernelorg-stable
mirror of https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

UAPI Changes: 

- Add madvise interface (Himal Prasad Ghimiray)
  - Add DRM_IOCTL_XE_VM_QUERY_MEMORY_RANGE_ATTRS to query VMA count and
    memory attributes (Himal Prasad Ghimiray)
  - Handle Firmware reported Hardware Errors notifying userspace with
    device wedged uevent (Riana Tauro)
 
 Cross-subsystem Changes:
 
  - Add a vendor-specific recovery method to drm device wedged uevent
    (Riana Tauro)
 
 Driver Changes:
  - Use same directory structure in debugfs as in sysfs (Michal Wajdeczko)
  - Cleanup and future-proof VRAM region initialization (Piotr Piórkowski)
  - Add G-states and PCIe link states to debugfs (Soham Purkait)
  - Cleanup eustall debug messages (Harish Chegondi)
  - Add SR-IOV support to restore Compression Control Surface (CCS) to
    Xe2 and later (Satyanarayana K V P)
  - Enable SR-IOV PF mode by default on supported platforms without
    needing CONFIG_DRM_XE_DEBUG and mark some platforms behind
    force_probe as supported (Michal Wajdeczko)
  - More targeted log messages (Michal Wajdeczko)
  - Cleanup STEER_SEMAPHORE/MCFG_MCR_SELECTOR usage (Nitin Gote)
  - Use common code to emit flush (Tvrtko Ursulin)
  - Add/extend more HW workarounds and tunings for Xe2 and Xe3
    (Sk Anirban, Tangudu Tilak Tirumalesh, Nitin Gote, Chaitanya Kumar Borah)
  - Add a generic dependency scheduler to help with TLB invalidations
    and future scenarios (Matthew Brost)
  - Use DRM scheduler for delayed GT TLB invalidations (Matthew Brost)
  - Error out on incorrect device use in configfs
    (Michal Wajdeczko, Lucas De Marchi)
  - Refactor configfs attributes (Michal Wajdeczko / Lucas De Marchi)
  - Allow configuring future VF devices via configfs (Michal Wajdeczko)
  - Implement some missing XeLP workarounds (Tvrtko Ursulin)
  - Generalize WA BB setup/emission and add support for
    mid context restore BB, aka indirect context (Tvrtko Ursulin)
  - Prepare the driver to expose mmio regions to userspace
    in future (Ilia Levi)
  - Add more GuC load error status codes (John Harrison)
  - Document DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING (Priyanka Dandamudi)
  - Disable CSC and RPM on VFs (Lukasz Laguna, Satyanarayana K V P)
  - Fix oops in xe_gem_fault with PREEMPT_RT (Maarten Lankhorst)
  - Skip LMTT update if no LMEM was provisioned (Michal Wajdeczko)
  - Add support to VF migration (Tomasz Lis)
  - Use a helper for guc_waklv_enable functions (Jonathan Cavitt)
  - Prepare GPU SVM for migration of THP (Francois Dugast)
  - Program LMTT directory pointer on all GTs within a tile
    (Piotr Piórkowski)
  - Rename XE_WA to XE_GT_WA to better convey its scope vs the device WAs
    (Matt Atwood)
  - Allow to match devices on PCI devid/vendorid only (Lucas De Marchi)
  - Improve PDE PAT index selection (Matthew Brost)
  - Consolidate ASID allocation in xe_vm_create() vs
    xe_vm_create_ioctl() (Piotr Piórkowski)
  - Resize VF BARS to max possible size according to number of VFs
    (Michał Winiarski)
  - Untangle vm_bind_ioctl cleanup order (Christoph Manszewski)
  - Start fixing usage of XE_PAGE_SIZE vs PAGE_SIZE to improve
    compatibility with non-x86 arch (Simon Richter)
  - Improve tile vs gt initialization order and accounting
    (Gustavo Sousa)
  - Extend WA kunit test to PTL
  - Ensure data is initialized before transferring to pcode
    (Stuart Summers)
  - Add PSMI support for HW validation (Lucas De Marchi,
    Vinay Belgaumkar, Badal Nilawar)
  - Improve xe_dma_buf test (Thomas Hellström, Marcin Bernatowicz)
  - Fix basename() usage in generator with !glibc (Carlos Llamas)
  - Ensure GT is in C0 during resumes (Xin Wang)
  - Add TLB invalidation abstraction (Matt Brost, Stuart Summers)
  - Make MI_TLB_INVALIDATE conditional on migrate (Matthew Auld)
  - Prepare xe_nvm to be initialized early for future use cases
    (Riana Tauro)
 -----BEGIN PGP SIGNATURE-----
 
 iQJNBAABCgA3FiEE6rM8lpABPHM5FqyDm6KlpjDL6lMFAmiyILoZHGx1Y2FzLmRl
 bWFyY2hpQGludGVsLmNvbQAKCRCboqWmMMvqU7teEACUioL2IQ7FCRSE8WJiP19O
 ziijaUbPkPy1um+njj+di2aUrsjGHF18PSRJgxwI73xay+hwE+Gtd3JrR6qc8g7l
 5q5epnwP0A3b30sEHKhm96qnUWxzyoi52P6uzx/cIQZ3MJS8OOUQL65FLxWgH0Ft
 ScuuDd6AH3OPRe8W5NgjyiCgXIWtS4PrHzYO7A1fxHIhNpKunrBBTnLuqTqZD+CM
 snNbJeaA+d2usCg11iF1/TGoZd9R3EuhEl+37CfoVWtQBgK4h5qkR0uyPO/PGblH
 YSYgjWYR7AyZghltD6RZ6NHoLaKbZ18Cf+tWJF8zgYY37APFPk2woIxb/Hgc1Fq7
 +ZetW0HEJgQVpytC+7qGucC7yFKTKbDQi/1bjMT9Odh+Ks9Ow5Vh0ObBWflKxFMd
 Ojrx2MqWqFMLtU9l2bY86VSrpe4DtyGHv56y1RumuzImAGKLTgHQsaQ9tarZVi3Z
 s30ah5wy6EmW6KIzUSgO2tBCK3cmpv77y4y02H5DhfHS9fFaacJrDe9JLybRr9p7
 APhYipKYvypv/db6HaVy0ItUzrPIkYtWA8fqyxXUTiURvqA2zkJXZnFYtJNSw1n5
 q3duTMDiGW9zQRqRs16urz80JIQpHTxeNk2uij7j2M88lRxGsQxRUGhHcLjvL8xn
 Q1biLu+pWToGQA+DvwTqAg==
 =gqoN
 -----END PGP SIGNATURE-----

Merge tag 'drm-xe-next-2025-08-29' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-next

UAPI Changes:
 - Add madvise interface (Himal Prasad Ghimiray)
 - Add DRM_IOCTL_XE_VM_QUERY_MEMORY_RANGE_ATTRS to query VMA count and
   memory attributes (Himal Prasad Ghimiray)
 - Handle Firmware reported Hardware Errors notifying userspace with
   device wedged uevent (Riana Tauro)

Cross-subsystem Changes:

 - Add a vendor-specific recovery method to drm device wedged uevent
   (Riana Tauro)

Driver Changes:
 - Use same directory structure in debugfs as in sysfs (Michal Wajdeczko)
 - Cleanup and future-proof VRAM region initialization (Piotr Piórkowski)
 - Add G-states and PCIe link states to debugfs (Soham Purkait)
 - Cleanup eustall debug messages (Harish Chegondi)
 - Add SR-IOV support to restore Compression Control Surface (CCS) to
   Xe2 and later (Satyanarayana K V P)
 - Enable SR-IOV PF mode by default on supported platforms without
   needing CONFIG_DRM_XE_DEBUG and mark some platforms behind
   force_probe as supported (Michal Wajdeczko)
 - More targeted log messages (Michal Wajdeczko)
 - Cleanup STEER_SEMAPHORE/MCFG_MCR_SELECTOR usage (Nitin Gote)
 - Use common code to emit flush (Tvrtko Ursulin)
 - Add/extend more HW workarounds and tunings for Xe2 and Xe3
   (Sk Anirban, Tangudu Tilak Tirumalesh, Nitin Gote, Chaitanya Kumar Borah)
 - Add a generic dependency scheduler to help with TLB invalidations
   and future scenarios (Matthew Brost)
 - Use DRM scheduler for delayed GT TLB invalidations (Matthew Brost)
 - Error out on incorrect device use in configfs
   (Michal Wajdeczko, Lucas De Marchi)
 - Refactor configfs attributes (Michal Wajdeczko / Lucas De Marchi)
 - Allow configuring future VF devices via configfs (Michal Wajdeczko)
 - Implement some missing XeLP workarounds (Tvrtko Ursulin)
 - Generalize WA BB setup/emission and add support for
   mid context restore BB, aka indirect context (Tvrtko Ursulin)
 - Prepare the driver to expose mmio regions to userspace
   in future (Ilia Levi)
 - Add more GuC load error status codes (John Harrison)
 - Document DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING (Priyanka Dandamudi)
 - Disable CSC and RPM on VFs (Lukasz Laguna, Satyanarayana K V P)
 - Fix oops in xe_gem_fault with PREEMPT_RT (Maarten Lankhorst)
 - Skip LMTT update if no LMEM was provisioned (Michal Wajdeczko)
 - Add support to VF migration (Tomasz Lis)
 - Use a helper for guc_waklv_enable functions (Jonathan Cavitt)
 - Prepare GPU SVM for migration of THP (Francois Dugast)
 - Program LMTT directory pointer on all GTs within a tile
   (Piotr Piórkowski)
 - Rename XE_WA to XE_GT_WA to better convey its scope vs the device WAs
   (Matt Atwood)
 - Allow to match devices on PCI devid/vendorid only (Lucas De Marchi)
 - Improve PDE PAT index selection (Matthew Brost)
 - Consolidate ASID allocation in xe_vm_create() vs
   xe_vm_create_ioctl() (Piotr Piórkowski)
 - Resize VF BARS to max possible size according to number of VFs
   (Michał Winiarski)
 - Untangle vm_bind_ioctl cleanup order (Christoph Manszewski)
 - Start fixing usage of XE_PAGE_SIZE vs PAGE_SIZE to improve
   compatibility with non-x86 arch (Simon Richter)
 - Improve tile vs gt initialization order and accounting
   (Gustavo Sousa)
 - Extend WA kunit test to PTL
 - Ensure data is initialized before transferring to pcode
   (Stuart Summers)
 - Add PSMI support for HW validation (Lucas De Marchi,
   Vinay Belgaumkar, Badal Nilawar)
 - Improve xe_dma_buf test (Thomas Hellström, Marcin Bernatowicz)
 - Fix basename() usage in generator with !glibc (Carlos Llamas)
 - Ensure GT is in C0 during resumes (Xin Wang)
 - Add TLB invalidation abstraction (Matt Brost, Stuart Summers)
 - Make MI_TLB_INVALIDATE conditional on migrate (Matthew Auld)
 - Prepare xe_nvm to be initialized early for future use cases
   (Riana Tauro)

Signed-off-by: Dave Airlie <airlied@redhat.com>

From: Lucas De Marchi <lucas.demarchi@intel.com>
Link: https://lore.kernel.org/r/nuejxdhnalyok7tzwkrj67dwjgdafwp4mhdejpyyqnrh4f2epq@nlldovuflnbx
This commit is contained in:
Dave Airlie 2025-09-02 10:05:55 +10:00
parent 14579a6f18 1047bd8279
commit 83631c7b1f
154 changed files with 7228 additions and 1877 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
Documentation/gpu/drm-uapi.rst
View File

@ -418,13 +418,12 @@ needed.
Recovery
--------
Current implementation defines three recovery methods, out of which, drivers
Current implementation defines four recovery methods, out of which, drivers
can use any one, multiple or none. Method(s) of choice will be sent in the
uevent environment as ``WEDGED=<method1>[,..,<methodN>]`` in order of less to
more side-effects. If driver is unsure about recovery or method is unknown
(like soft/hard system reboot, firmware flashing, physical device replacement
or any other procedure which can't be attempted on the fly), ``WEDGED=unknown``
will be sent instead.
more side-effects. See the section `Vendor Specific Recovery`_
for ``WEDGED=vendor-specific``. If driver is unsure about recovery or
method is unknown, ``WEDGED=unknown`` will be sent instead.
Userspace consumers can parse this event and attempt recovery as per the
following expectations.
@ -435,6 +434,7 @@ following expectations.
none optional telemetry collection
rebind unbind + bind driver
bus-reset unbind + bus reset/re-enumeration + bind
vendor-specific vendor specific recovery method
unknown consumer policy
=============== ========================================
@ -446,6 +446,35 @@ telemetry information (devcoredump, syslog). This is useful because the first
hang is usually the most critical one which can result in consequential hangs or
complete wedging.
Vendor Specific Recovery
------------------------
When ``WEDGED=vendor-specific`` is sent, it indicates that the device requires
a recovery procedure specific to the hardware vendor and is not one of the
standardized approaches.
``WEDGED=vendor-specific`` may be used to indicate different cases within a
single vendor driver, each requiring a distinct recovery procedure.
In such scenarios, the vendor driver must provide comprehensive documentation
that describes each case, include additional hints to identify specific case and
outline the corresponding recovery procedure. The documentation includes:
Case - A list of all cases that sends the ``WEDGED=vendor-specific`` recovery method.
Hints - Additional Information to assist the userspace consumer in identifying and
differentiating between different cases. This can be exposed through sysfs, debugfs,
traces, dmesg etc.
Recovery Procedure - Clear instructions and guidance for recovering each case.
This may include userspace scripts, tools needed for the recovery procedure.
It is the responsibility of the admin/userspace consumer to identify the case and
verify additional identification hints before attempting a recovery procedure.
Example: If the device uses the Xe driver, then userspace consumer should refer to
:ref:`Xe Device Wedging <xe-device-wedging>` for the detailed documentation.
Task information
----------------
@ -472,8 +501,12 @@ erroring out, all device memory should be unmapped and file descriptors should
be closed to prevent leaks or undefined behaviour. The idea here is to clear the
device of all user context beforehand and set the stage for a clean recovery.
Example
-------
For ``WEDGED=vendor-specific`` recovery method, it is the responsibility of the
consumer to check the driver documentation and the usecase before attempting
a recovery.
Example - rebind
----------------
Udev rule::

1
Documentation/gpu/xe/index.rst
View File

@ -25,5 +25,6 @@ DG2, etc is provided to prototype the driver.
xe_tile
xe_debugging
xe_devcoredump
xe_device
xe-drm-usage-stats.rst
xe_configfs

10
Documentation/gpu/xe/xe_device.rst Normal file
View File

@ -0,0 +1,10 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
.. _xe-device-wedging:
==================
Xe Device Wedging
==================
.. kernel-doc:: drivers/gpu/drm/xe/xe_device.c
:doc: Xe Device Wedging

6
Documentation/gpu/xe/xe_pcode.rst
View File

@ -13,9 +13,11 @@ Internal API
.. kernel-doc:: drivers/gpu/drm/xe/xe_pcode.c
:internal:
.. _xe-survivability-mode:
==================
Boot Survivability
Survivability Mode
==================
.. kernel-doc:: drivers/gpu/drm/xe/xe_survivability_mode.c
:doc: Xe Boot Survivability
:doc: Survivability Mode

2
drivers/gpu/drm/drm_drv.c
View File

@ -532,6 +532,8 @@ static const char *drm_get_wedge_recovery(unsigned int opt)
return "rebind";
case DRM_WEDGE_RECOVERY_BUS_RESET:
return "bus-reset";
case DRM_WEDGE_RECOVERY_VENDOR:
return "vendor-specific";
default:
return NULL;
}

4
drivers/gpu/drm/drm_gpusvm.c
View File

@ -975,7 +975,7 @@ static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
};
for (i = 0, j = 0; i < npages; j++) {
struct drm_pagemap_device_addr *addr = &range->dma_addr[j];
struct drm_pagemap_addr *addr = &range->dma_addr[j];
if (addr->proto == DRM_INTERCONNECT_SYSTEM)
dma_unmap_page(dev,
@ -1328,7 +1328,7 @@ map_pages:
goto err_unmap;
}
range->dma_addr[j] = drm_pagemap_device_addr_encode
range->dma_addr[j] = drm_pagemap_addr_encode
(addr, DRM_INTERCONNECT_SYSTEM, order,
DMA_BIDIRECTIONAL);
}

142
drivers/gpu/drm/drm_pagemap.c
View File

@ -202,7 +202,7 @@ static void drm_pagemap_get_devmem_page(struct page *page,
/**
* drm_pagemap_migrate_map_pages() - Map migration pages for GPU SVM migration
* @dev: The device for which the pages are being mapped
* @dma_addr: Array to store DMA addresses corresponding to mapped pages
* @pagemap_addr: Array to store DMA information corresponding to mapped pages
* @migrate_pfn: Array of migrate page frame numbers to map
* @npages: Number of pages to map
* @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
@ -215,25 +215,39 @@ static void drm_pagemap_get_devmem_page(struct page *page,
* Returns: 0 on success, -EFAULT if an error occurs during mapping.
*/
static int drm_pagemap_migrate_map_pages(struct device *dev,
dma_addr_t *dma_addr,
struct drm_pagemap_addr *pagemap_addr,
unsigned long *migrate_pfn,
unsigned long npages,
enum dma_data_direction dir)
{
unsigned long i;
for (i = 0; i < npages; ++i) {
for (i = 0; i < npages;) {
struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
dma_addr_t dma_addr;
struct folio *folio;
unsigned int order = 0;
if (!page)
continue;
goto next;
if (WARN_ON_ONCE(is_zone_device_page(page)))
return -EFAULT;
dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
if (dma_mapping_error(dev, dma_addr[i]))
folio = page_folio(page);
order = folio_order(folio);
dma_addr = dma_map_page(dev, page, 0, page_size(page), dir);
if (dma_mapping_error(dev, dma_addr))
return -EFAULT;
pagemap_addr[i] =
drm_pagemap_addr_encode(dma_addr,
DRM_INTERCONNECT_SYSTEM,
order, dir);
next:
i += NR_PAGES(order);
}
return 0;
@ -242,7 +256,7 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
/**
* drm_pagemap_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
* @dev: The device for which the pages were mapped
* @dma_addr: Array of DMA addresses corresponding to mapped pages
* @pagemap_addr: Array of DMA information corresponding to mapped pages
* @npages: Number of pages to unmap
* @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
*
@ -251,17 +265,20 @@ static int drm_pagemap_migrate_map_pages(struct device *dev,
* if it's valid and not already unmapped, and unmaps the corresponding page.
*/
static void drm_pagemap_migrate_unmap_pages(struct device *dev,
dma_addr_t *dma_addr,
struct drm_pagemap_addr *pagemap_addr,
unsigned long npages,
enum dma_data_direction dir)
{
unsigned long i;
for (i = 0; i < npages; ++i) {
if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
continue;
for (i = 0; i < npages;) {
if (!pagemap_addr[i].addr || dma_mapping_error(dev, pagemap_addr[i].addr))
goto next;
dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
dma_unmap_page(dev, pagemap_addr[i].addr, PAGE_SIZE << pagemap_addr[i].order, dir);
next:
i += NR_PAGES(pagemap_addr[i].order);
}
}
@ -314,7 +331,7 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
struct vm_area_struct *vas;
struct drm_pagemap_zdd *zdd = NULL;
struct page **pages;
dma_addr_t *dma_addr;
struct drm_pagemap_addr *pagemap_addr;
void *buf;
int err;
@ -340,14 +357,14 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
goto err_out;
}
buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*pagemap_addr) +
sizeof(*pages), GFP_KERNEL);
if (!buf) {
err = -ENOMEM;
goto err_out;
}
dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
pagemap_addr = buf + (2 * sizeof(*migrate.src) * npages);
pages = buf + (2 * sizeof(*migrate.src) + sizeof(*pagemap_addr)) * npages;
zdd = drm_pagemap_zdd_alloc(pgmap_owner);
if (!zdd) {
@ -377,8 +394,9 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
if (err)
goto err_finalize;
err = drm_pagemap_migrate_map_pages(devmem_allocation->dev, dma_addr,
err = drm_pagemap_migrate_map_pages(devmem_allocation->dev, pagemap_addr,
migrate.src, npages, DMA_TO_DEVICE);
if (err)
goto err_finalize;
@ -390,7 +408,7 @@ int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
drm_pagemap_get_devmem_page(page, zdd);
}
err = ops->copy_to_devmem(pages, dma_addr, npages);
err = ops->copy_to_devmem(pages, pagemap_addr, npages);
if (err)
goto err_finalize;
@ -404,7 +422,7 @@ err_finalize:
drm_pagemap_migration_unlock_put_pages(npages, migrate.dst);
migrate_vma_pages(&migrate);
migrate_vma_finalize(&migrate);
drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, dma_addr, npages,
drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, npages,
DMA_TO_DEVICE);
err_free:
if (zdd)
@ -442,54 +460,80 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas,
{
unsigned long i;
for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
struct page *page, *src_page;
for (i = 0; i < npages;) {
struct page *page = NULL, *src_page;
struct folio *folio;
unsigned int order = 0;
if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
continue;
goto next;
src_page = migrate_pfn_to_page(src_mpfn[i]);
if (!src_page)
continue;
goto next;
if (fault_page) {
if (src_page->zone_device_data !=
fault_page->zone_device_data)
continue;
goto next;
}
if (vas)
page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
else
page = alloc_page(GFP_HIGHUSER);
order = folio_order(page_folio(src_page));
if (!page)
/* TODO: Support fallback to single pages if THP allocation fails */
if (vas)
folio = vma_alloc_folio(GFP_HIGHUSER, order, vas, addr);
else
folio = folio_alloc(GFP_HIGHUSER, order);
if (!folio)
goto free_pages;
page = folio_page(folio, 0);
mpfn[i] = migrate_pfn(page_to_pfn(page));
next:
if (page)
addr += page_size(page);
else
addr += PAGE_SIZE;
i += NR_PAGES(order);
}
for (i = 0; i < npages; ++i) {
for (i = 0; i < npages;) {
struct page *page = migrate_pfn_to_page(mpfn[i]);
unsigned int order = 0;
if (!page)
continue;
goto next_lock;
WARN_ON_ONCE(!trylock_page(page));
++*mpages;
WARN_ON_ONCE(!folio_trylock(page_folio(page)));
order = folio_order(page_folio(page));
*mpages += NR_PAGES(order);
next_lock:
i += NR_PAGES(order);
}
return 0;
free_pages:
for (i = 0; i < npages; ++i) {
for (i = 0; i < npages;) {
struct page *page = migrate_pfn_to_page(mpfn[i]);
unsigned int order = 0;
if (!page)
continue;
goto next_put;
put_page(page);
mpfn[i] = 0;
order = folio_order(page_folio(page));
next_put:
i += NR_PAGES(order);
}
return -ENOMEM;
}
@ -509,7 +553,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation)
unsigned long npages, mpages = 0;
struct page **pages;
unsigned long *src, *dst;
dma_addr_t *dma_addr;
struct drm_pagemap_addr *pagemap_addr;
void *buf;
int i, err = 0;
unsigned int retry_count = 2;
@ -520,7 +564,7 @@ retry:
if (!mmget_not_zero(devmem_allocation->mm))
return -EFAULT;
buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*pagemap_addr) +
sizeof(*pages), GFP_KERNEL);
if (!buf) {
err = -ENOMEM;
@ -528,8 +572,8 @@ retry:
}
src = buf;
dst = buf + (sizeof(*src) * npages);
dma_addr = buf + (2 * sizeof(*src) * npages);
pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
pagemap_addr = buf + (2 * sizeof(*src) * npages);
pages = buf + (2 * sizeof(*src) + sizeof(*pagemap_addr)) * npages;
err = ops->populate_devmem_pfn(devmem_allocation, npages, src);
if (err)
@ -544,7 +588,7 @@ retry:
if (err || !mpages)
goto err_finalize;
err = drm_pagemap_migrate_map_pages(devmem_allocation->dev, dma_addr,
err = drm_pagemap_migrate_map_pages(devmem_allocation->dev, pagemap_addr,
dst, npages, DMA_FROM_DEVICE);
if (err)
goto err_finalize;
@ -552,7 +596,7 @@ retry:
for (i = 0; i < npages; ++i)
pages[i] = migrate_pfn_to_page(src[i]);
err = ops->copy_to_ram(pages, dma_addr, npages);
err = ops->copy_to_ram(pages, pagemap_addr, npages);
if (err)
goto err_finalize;
@ -561,7 +605,7 @@ err_finalize:
drm_pagemap_migration_unlock_put_pages(npages, dst);
migrate_device_pages(src, dst, npages);
migrate_device_finalize(src, dst, npages);
drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, dma_addr, npages,
drm_pagemap_migrate_unmap_pages(devmem_allocation->dev, pagemap_addr, npages,
DMA_FROM_DEVICE);
err_free:
kvfree(buf);
@ -612,7 +656,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
struct device *dev = NULL;
unsigned long npages, mpages = 0;
struct page **pages;
dma_addr_t *dma_addr;
struct drm_pagemap_addr *pagemap_addr;
unsigned long start, end;
void *buf;
int i, err = 0;
@ -637,14 +681,14 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
migrate.end = end;
npages = npages_in_range(start, end);
buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*pagemap_addr) +
sizeof(*pages), GFP_KERNEL);
if (!buf) {
err = -ENOMEM;
goto err_out;
}
dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
pagemap_addr = buf + (2 * sizeof(*migrate.src) * npages);
pages = buf + (2 * sizeof(*migrate.src) + sizeof(*pagemap_addr)) * npages;
migrate.vma = vas;
migrate.src = buf;
@ -680,7 +724,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
if (err)
goto err_finalize;
err = drm_pagemap_migrate_map_pages(dev, dma_addr, migrate.dst, npages,
err = drm_pagemap_migrate_map_pages(dev, pagemap_addr, migrate.dst, npages,
DMA_FROM_DEVICE);
if (err)
goto err_finalize;
@ -688,7 +732,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
for (i = 0; i < npages; ++i)
pages[i] = migrate_pfn_to_page(migrate.src[i]);
err = ops->copy_to_ram(pages, dma_addr, npages);
err = ops->copy_to_ram(pages, pagemap_addr, npages);
if (err)
goto err_finalize;
@ -698,7 +742,7 @@ err_finalize:
migrate_vma_pages(&migrate);
migrate_vma_finalize(&migrate);
if (dev)
drm_pagemap_migrate_unmap_pages(dev, dma_addr, npages,
drm_pagemap_migrate_unmap_pages(dev, pagemap_addr, npages,
DMA_FROM_DEVICE);
err_free:
kvfree(buf);

10
drivers/gpu/drm/xe/Makefile
View File

@ -35,6 +35,7 @@ $(obj)/generated/%_device_wa_oob.c $(obj)/generated/%_device_wa_oob.h: $(obj)/xe
xe-y += xe_bb.o \
xe_bo.o \
xe_bo_evict.o \
xe_dep_scheduler.o \
xe_devcoredump.o \
xe_device.o \
xe_device_sysfs.o \
@ -60,7 +61,6 @@ xe-y += xe_bb.o \
xe_gt_pagefault.o \
xe_gt_sysfs.o \
xe_gt_throttle.o \
xe_gt_tlb_invalidation.o \
xe_gt_topology.o \
xe_guc.o \
xe_guc_ads.o \
@ -75,16 +75,19 @@ xe-y += xe_bb.o \
xe_guc_log.o \
xe_guc_pc.o \
xe_guc_submit.o \
xe_guc_tlb_inval.o \
xe_heci_gsc.o \
xe_huc.o \
xe_hw_engine.o \
xe_hw_engine_class_sysfs.o \
xe_hw_engine_group.o \
xe_hw_error.o \
xe_hw_fence.o \
xe_irq.o \
xe_lrc.o \
xe_migrate.o \
xe_mmio.o \
xe_mmio_gem.o \
xe_mocs.o \
xe_module.o \
xe_nvm.o \
@ -95,6 +98,7 @@ xe-y += xe_bb.o \
xe_pcode.o \
xe_pm.o \
xe_preempt_fence.o \
xe_psmi.o \
xe_pt.o \
xe_pt_walk.o \
xe_pxp.o \
@ -114,6 +118,8 @@ xe-y += xe_bb.o \
xe_sync.o \
xe_tile.o \
xe_tile_sysfs.o \
xe_tlb_inval.o \
xe_tlb_inval_job.o \
xe_trace.o \
xe_trace_bo.o \
xe_trace_guc.o \
@ -125,6 +131,7 @@ xe-y += xe_bb.o \
xe_uc.o \
xe_uc_fw.o \
xe_vm.o \
xe_vm_madvise.o \
xe_vram.o \
xe_vram_freq.o \
xe_vsec.o \
@ -149,6 +156,7 @@ xe-y += \
xe_memirq.o \
xe_sriov.o \
xe_sriov_vf.o \
xe_sriov_vf_ccs.o \
xe_tile_sriov_vf.o
xe-$(CONFIG_PCI_IOV) += \

8
drivers/gpu/drm/xe/abi/guc_actions_abi.h
View File

@ -193,6 +193,14 @@ enum xe_guc_register_context_multi_lrc_param_offsets {
XE_GUC_REGISTER_CONTEXT_MULTI_LRC_MSG_MIN_LEN = 11,
};
enum xe_guc_context_wq_item_offsets {
XE_GUC_CONTEXT_WQ_HEADER_DATA_0_TYPE_LEN = 0,
XE_GUC_CONTEXT_WQ_EL_INFO_DATA_1_CTX_DESC_LOW,
XE_GUC_CONTEXT_WQ_EL_INFO_DATA_2_GUCCTX_RINGTAIL_FREEZEPOCS,
XE_GUC_CONTEXT_WQ_EL_INFO_DATA_3_WI_FENCE_ID,
XE_GUC_CONTEXT_WQ_EL_CHILD_LIST_DATA_4_RINGTAIL,
};
enum xe_guc_report_status {
XE_GUC_REPORT_STATUS_UNKNOWN = 0x0,
XE_GUC_REPORT_STATUS_ACKED = 0x1,

3
drivers/gpu/drm/xe/abi/guc_errors_abi.h
View File

@ -63,6 +63,7 @@ enum xe_guc_load_status {
XE_GUC_LOAD_STATUS_HWCONFIG_START = 0x05,
XE_GUC_LOAD_STATUS_HWCONFIG_DONE = 0x06,
XE_GUC_LOAD_STATUS_HWCONFIG_ERROR = 0x07,
XE_GUC_LOAD_STATUS_BOOTROM_VERSION_MISMATCH = 0x08,
XE_GUC_LOAD_STATUS_GDT_DONE = 0x10,
XE_GUC_LOAD_STATUS_IDT_DONE = 0x20,
XE_GUC_LOAD_STATUS_LAPIC_DONE = 0x30,
@ -75,6 +76,8 @@ enum xe_guc_load_status {
XE_GUC_LOAD_STATUS_INVALID_INIT_DATA_RANGE_START,
XE_GUC_LOAD_STATUS_MPU_DATA_INVALID = 0x73,
XE_GUC_LOAD_STATUS_INIT_MMIO_SAVE_RESTORE_INVALID = 0x74,
XE_GUC_LOAD_STATUS_KLV_WORKAROUND_INIT_ERROR = 0x75,
XE_GUC_LOAD_STATUS_INVALID_FTR_FLAG = 0x76,
XE_GUC_LOAD_STATUS_INVALID_INIT_DATA_RANGE_END,
XE_GUC_LOAD_STATUS_READY = 0xF0,

2
drivers/gpu/drm/xe/abi/guc_klvs_abi.h
View File

@ -390,12 +390,14 @@ enum {
*/
enum xe_guc_klv_ids {
GUC_WORKAROUND_KLV_BLOCK_INTERRUPTS_WHEN_MGSR_BLOCKED = 0x9002,
GUC_WORKAROUND_KLV_DISABLE_PSMI_INTERRUPTS_AT_C6_ENTRY_RESTORE_AT_EXIT = 0x9004,
GUC_WORKAROUND_KLV_ID_GAM_PFQ_SHADOW_TAIL_POLLING = 0x9005,
GUC_WORKAROUND_KLV_ID_DISABLE_MTP_DURING_ASYNC_COMPUTE = 0x9007,
GUC_WA_KLV_NP_RD_WRITE_TO_CLEAR_RCSM_AT_CGP_LATE_RESTORE = 0x9008,
GUC_WORKAROUND_KLV_ID_BACK_TO_BACK_RCS_ENGINE_RESET = 0x9009,
GUC_WA_KLV_WAKE_POWER_DOMAINS_FOR_OUTBOUND_MMIO = 0x900a,
GUC_WA_KLV_RESET_BB_STACK_PTR_ON_VF_SWITCH = 0x900b,
GUC_WA_KLV_RESTORE_UNSAVED_MEDIA_CONTROL_REG = 0x900c,
};
#endif

2
drivers/gpu/drm/xe/display/intel_fbdev_fb.c
View File

@ -41,7 +41,7 @@ struct intel_framebuffer *intel_fbdev_fb_alloc(struct drm_fb_helper *helper,
size = PAGE_ALIGN(size);
obj = ERR_PTR(-ENODEV);
if (!IS_DGFX(xe) && !XE_WA(xe_root_mmio_gt(xe), 22019338487_display)) {
if (!IS_DGFX(xe) && !XE_GT_WA(xe_root_mmio_gt(xe), 22019338487_display)) {
obj = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe),
NULL, size,
ttm_bo_type_kernel, XE_BO_FLAG_SCANOUT |

2
drivers/gpu/drm/xe/display/xe_display_wa.c
View File

@ -14,5 +14,5 @@ bool intel_display_needs_wa_16023588340(struct intel_display *display)
{
struct xe_device *xe = to_xe_device(display->drm);
return XE_WA(xe_root_mmio_gt(xe), 16023588340);
return XE_GT_WA(xe_root_mmio_gt(xe), 16023588340);
}

5
drivers/gpu/drm/xe/display/xe_fb_pin.c
View File

@ -16,6 +16,7 @@
#include "xe_device.h"
#include "xe_ggtt.h"
#include "xe_pm.h"
#include "xe_vram_types.h"
static void
write_dpt_rotated(struct xe_bo *bo, struct iosys_map *map, u32 *dpt_ofs, u32 bo_ofs,
@ -289,7 +290,7 @@ static struct i915_vma *__xe_pin_fb_vma(const struct intel_framebuffer *fb,
if (IS_DGFX(to_xe_device(bo->ttm.base.dev)) &&
intel_fb_rc_ccs_cc_plane(&fb->base) >= 0 &&
!(bo->flags & XE_BO_FLAG_NEEDS_CPU_ACCESS)) {
struct xe_tile *tile = xe_device_get_root_tile(xe);
struct xe_vram_region *vram = xe_device_get_root_tile(xe)->mem.vram;
/*
* If we need to able to access the clear-color value stored in
@ -297,7 +298,7 @@ static struct i915_vma *__xe_pin_fb_vma(const struct intel_framebuffer *fb,
* accessible. This is important on small-bar systems where
* only some subset of VRAM is CPU accessible.
*/
if (tile->mem.vram.io_size < tile->mem.vram.usable_size) {
if (xe_vram_region_io_size(vram) < xe_vram_region_usable_size(vram)) {
ret = -EINVAL;
goto err;
}

5
drivers/gpu/drm/xe/display/xe_plane_initial.c
View File

@ -21,6 +21,7 @@
#include "intel_plane.h"
#include "intel_plane_initial.h"
#include "xe_bo.h"
#include "xe_vram_types.h"
#include "xe_wa.h"
#include <generated/xe_wa_oob.h>
@ -103,7 +104,7 @@ initial_plane_bo(struct xe_device *xe,
* We don't currently expect this to ever be placed in the
* stolen portion.
*/
if (phys_base >= tile0->mem.vram.usable_size) {
if (phys_base >= xe_vram_region_usable_size(tile0->mem.vram)) {
drm_err(&xe->drm,
"Initial plane programming using invalid range, phys_base=%pa\n",
&phys_base);
@ -121,7 +122,7 @@ initial_plane_bo(struct xe_device *xe,
phys_base = base;
flags |= XE_BO_FLAG_STOLEN;
if (XE_WA(xe_root_mmio_gt(xe), 22019338487_display))
if (XE_GT_WA(xe_root_mmio_gt(xe), 22019338487_display))
return NULL;
/*

1
drivers/gpu/drm/xe/instructions/xe_mi_commands.h
View File

@ -65,6 +65,7 @@
#define MI_LOAD_REGISTER_MEM (__MI_INSTR(0x29) | XE_INSTR_NUM_DW(4))
#define MI_LRM_USE_GGTT REG_BIT(22)
#define MI_LRM_ASYNC REG_BIT(21)
#define MI_LOAD_REGISTER_REG (__MI_INSTR(0x2a) | XE_INSTR_NUM_DW(3))
#define MI_LRR_DST_CS_MMIO REG_BIT(19)

3
drivers/gpu/drm/xe/regs/xe_engine_regs.h
View File

@ -111,6 +111,9 @@
#define PPHWSP_CSB_AND_TIMESTAMP_REPORT_DIS REG_BIT(14)
#define CS_PRIORITY_MEM_READ REG_BIT(7)
#define CS_DEBUG_MODE2(base) XE_REG((base) + 0xd8, XE_REG_OPTION_MASKED)
#define INSTRUCTION_STATE_CACHE_INVALIDATE REG_BIT(6)
#define FF_SLICE_CS_CHICKEN1(base) XE_REG((base) + 0xe0, XE_REG_OPTION_MASKED)
#define FFSC_PERCTX_PREEMPT_CTRL REG_BIT(14)

2
drivers/gpu/drm/xe/regs/xe_gsc_regs.h
View File

@ -13,6 +13,8 @@
/* Definitions of GSC H/W registers, bits, etc */
#define BMG_GSC_HECI1_BASE 0x373000
#define MTL_GSC_HECI1_BASE 0x00116000
#define MTL_GSC_HECI2_BASE 0x00117000

2
drivers/gpu/drm/xe/regs/xe_gt_regs.h
View File

@ -42,7 +42,7 @@
#define FORCEWAKE_ACK_GSC XE_REG(0xdf8)
#define FORCEWAKE_ACK_GT_MTL XE_REG(0xdfc)
#define MCFG_MCR_SELECTOR XE_REG(0xfd0)
#define STEER_SEMAPHORE XE_REG(0xfd0)
#define MTL_MCR_SELECTOR XE_REG(0xfd4)
#define SF_MCR_SELECTOR XE_REG(0xfd8)
#define MCR_SELECTOR XE_REG(0xfdc)

20
drivers/gpu/drm/xe/regs/xe_hw_error_regs.h Normal file
View File

@ -0,0 +1,20 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef _XE_HW_ERROR_REGS_H_
#define _XE_HW_ERROR_REGS_H_
#define HEC_UNCORR_ERR_STATUS(base) XE_REG((base) + 0x118)
#define UNCORR_FW_REPORTED_ERR BIT(6)
#define HEC_UNCORR_FW_ERR_DW0(base) XE_REG((base) + 0x124)
#define DEV_ERR_STAT_NONFATAL 0x100178
#define DEV_ERR_STAT_CORRECTABLE 0x10017c
#define DEV_ERR_STAT_REG(x) XE_REG(_PICK_EVEN((x), \
DEV_ERR_STAT_CORRECTABLE, \
DEV_ERR_STAT_NONFATAL))
#define XE_CSC_ERROR BIT(17)
#endif

1
drivers/gpu/drm/xe/regs/xe_irq_regs.h
View File

@ -18,6 +18,7 @@
#define GFX_MSTR_IRQ XE_REG(0x190010, XE_REG_OPTION_VF)
#define MASTER_IRQ REG_BIT(31)
#define GU_MISC_IRQ REG_BIT(29)
#define ERROR_IRQ(x) REG_BIT(26 + (x))
#define DISPLAY_IRQ REG_BIT(16)
#define I2C_IRQ REG_BIT(12)
#define GT_DW_IRQ(x) REG_BIT(x)

10
drivers/gpu/drm/xe/regs/xe_pmt.h
View File

@ -21,4 +21,14 @@
#define SG_REMAP_INDEX1 XE_REG(SOC_BASE + 0x08)
#define SG_REMAP_BITS REG_GENMASK(31, 24)
#define BMG_MODS_RESIDENCY_OFFSET (0x4D0)
#define BMG_G2_RESIDENCY_OFFSET (0x530)
#define BMG_G6_RESIDENCY_OFFSET (0x538)
#define BMG_G8_RESIDENCY_OFFSET (0x540)
#define BMG_G10_RESIDENCY_OFFSET (0x548)
#define BMG_PCIE_LINK_L0_RESIDENCY_OFFSET (0x570)
#define BMG_PCIE_LINK_L1_RESIDENCY_OFFSET (0x578)
#define BMG_PCIE_LINK_L1_2_RESIDENCY_OFFSET (0x580)
#endif

13
drivers/gpu/drm/xe/tests/xe_dma_buf.c
View File

@ -57,16 +57,12 @@ static void check_residency(struct kunit *test, struct xe_bo *exported,
return;
/*
* Evict exporter. Note that the gem object dma_buf member isn't
* set from xe_gem_prime_export(), and it's needed for the move_notify()
* functionality, so hack that up here. Evicting the exported bo will
* Evict exporter. Evicting the exported bo will
* evict also the imported bo through the move_notify() functionality if
* importer is on a different device. If they're on the same device,
* the exporter and the importer should be the same bo.
*/
swap(exported->ttm.base.dma_buf, dmabuf);
ret = xe_bo_evict(exported);
swap(exported->ttm.base.dma_buf, dmabuf);
if (ret) {
if (ret != -EINTR && ret != -ERESTARTSYS)
KUNIT_FAIL(test, "Evicting exporter failed with err=%d.\n",
@ -139,6 +135,7 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
PTR_ERR(dmabuf));
goto out;
}
bo->ttm.base.dma_buf = dmabuf;
import = xe_gem_prime_import(&xe->drm, dmabuf);
if (!IS_ERR(import)) {
@ -186,6 +183,7 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
KUNIT_FAIL(test, "dynamic p2p attachment failed with err=%ld\n",
PTR_ERR(import));
}
bo->ttm.base.dma_buf = NULL;
dma_buf_put(dmabuf);
out:
drm_gem_object_put(&bo->ttm.base);
@ -206,7 +204,7 @@ static const struct dma_buf_attach_ops nop2p_attach_ops = {
static const struct dma_buf_test_params test_params[] = {
{.mem_mask = XE_BO_FLAG_VRAM0,
.attach_ops = &xe_dma_buf_attach_ops},
{.mem_mask = XE_BO_FLAG_VRAM0,
{.mem_mask = XE_BO_FLAG_VRAM0 | XE_BO_FLAG_NEEDS_CPU_ACCESS,
.attach_ops = &xe_dma_buf_attach_ops,
.force_different_devices = true},
@ -238,7 +236,8 @@ static const struct dma_buf_test_params test_params[] = {
{.mem_mask = XE_BO_FLAG_SYSTEM | XE_BO_FLAG_VRAM0,
.attach_ops = &xe_dma_buf_attach_ops},
{.mem_mask = XE_BO_FLAG_SYSTEM | XE_BO_FLAG_VRAM0,
{.mem_mask = XE_BO_FLAG_SYSTEM | XE_BO_FLAG_VRAM0 |
XE_BO_FLAG_NEEDS_CPU_ACCESS,
.attach_ops = &xe_dma_buf_attach_ops,
.force_different_devices = true},

7
drivers/gpu/drm/xe/tests/xe_pci.c
View File

@ -101,6 +101,11 @@ static void fake_read_gmdid(struct xe_device *xe, enum xe_gmdid_type type,
}
}
static void fake_xe_info_probe_tile_count(struct xe_device *xe)
{
/* Nothing to do, just use the statically defined value. */
}
int xe_pci_fake_device_init(struct xe_device *xe)
{
struct kunit *test = kunit_get_current_test();
@ -138,6 +143,8 @@ done:
data->sriov_mode : XE_SRIOV_MODE_NONE;
kunit_activate_static_stub(test, read_gmdid, fake_read_gmdid);
kunit_activate_static_stub(test, xe_info_probe_tile_count,
fake_xe_info_probe_tile_count);
xe_info_init_early(xe, desc, subplatform_desc);
xe_info_init(xe, desc);

1
drivers/gpu/drm/xe/tests/xe_wa_test.c
View File

@ -75,6 +75,7 @@ static const struct platform_test_case cases[] = {
GMDID_CASE(LUNARLAKE, 2004, A0, 2000, A0),
GMDID_CASE(LUNARLAKE, 2004, B0, 2000, A0),
GMDID_CASE(BATTLEMAGE, 2001, A0, 1301, A1),
GMDID_CASE(PANTHERLAKE, 3000, A0, 3000, A0),
};
static void platform_desc(const struct platform_test_case *t, char *desc)

4
drivers/gpu/drm/xe/xe_assert.h
View File

@ -12,6 +12,7 @@
#include "xe_gt_types.h"
#include "xe_step.h"
#include "xe_vram.h"
/**
* DOC: Xe Asserts
@ -145,7 +146,8 @@
const struct xe_tile *__tile = (tile); \
char __buf[10] __maybe_unused; \
xe_assert_msg(tile_to_xe(__tile), condition, "tile: %u VRAM %s\n" msg, \
__tile->id, ({ string_get_size(__tile->mem.vram.actual_physical_size, 1, \
__tile->id, ({ string_get_size( \
xe_vram_region_actual_physical_size(__tile->mem.vram), 1, \
STRING_UNITS_2, __buf, sizeof(__buf)); __buf; }), ## arg); \
})

35
drivers/gpu/drm/xe/xe_bb.c
View File

@ -60,6 +60,41 @@ err:
return ERR_PTR(err);
}
struct xe_bb *xe_bb_ccs_new(struct xe_gt *gt, u32 dwords,
enum xe_sriov_vf_ccs_rw_ctxs ctx_id)
{
struct xe_bb *bb = kmalloc(sizeof(*bb), GFP_KERNEL);
struct xe_tile *tile = gt_to_tile(gt);
struct xe_sa_manager *bb_pool;
int err;
if (!bb)
return ERR_PTR(-ENOMEM);
/*
* We need to allocate space for the requested number of dwords &
* one additional MI_BATCH_BUFFER_END dword. Since the whole SA
* is submitted to HW, we need to make sure that the last instruction
* is not over written when the last chunk of SA is allocated for BB.
* So, this extra DW acts as a guard here.
*/
bb_pool = tile->sriov.vf.ccs[ctx_id].mem.ccs_bb_pool;
bb->bo = xe_sa_bo_new(bb_pool, 4 * (dwords + 1));
if (IS_ERR(bb->bo)) {
err = PTR_ERR(bb->bo);
goto err;
}
bb->cs = xe_sa_bo_cpu_addr(bb->bo);
bb->len = 0;
return bb;
err:
kfree(bb);
return ERR_PTR(err);
}
static struct xe_sched_job *
__xe_bb_create_job(struct xe_exec_queue *q, struct xe_bb *bb, u64 *addr)
{

3
drivers/gpu/drm/xe/xe_bb.h
View File

@ -13,8 +13,11 @@ struct dma_fence;
struct xe_gt;
struct xe_exec_queue;
struct xe_sched_job;
enum xe_sriov_vf_ccs_rw_ctxs;
struct xe_bb *xe_bb_new(struct xe_gt *gt, u32 dwords, bool usm);
struct xe_bb *xe_bb_ccs_new(struct xe_gt *gt, u32 dwords,
enum xe_sriov_vf_ccs_rw_ctxs ctx_id);
struct xe_sched_job *xe_bb_create_job(struct xe_exec_queue *q,
struct xe_bb *bb);
struct xe_sched_job *xe_bb_create_migration_job(struct xe_exec_queue *q,

94
drivers/gpu/drm/xe/xe_bo.c
View File

@ -33,9 +33,11 @@
#include "xe_pxp.h"
#include "xe_res_cursor.h"
#include "xe_shrinker.h"
#include "xe_sriov_vf_ccs.h"
#include "xe_trace_bo.h"
#include "xe_ttm_stolen_mgr.h"
#include "xe_vm.h"
#include "xe_vram_types.h"
const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES] = {
[XE_PL_SYSTEM] = "system",
@ -198,6 +200,8 @@ static bool force_contiguous(u32 bo_flags)
else if (bo_flags & XE_BO_FLAG_PINNED &&
!(bo_flags & XE_BO_FLAG_PINNED_LATE_RESTORE))
return true; /* needs vmap */
else if (bo_flags & XE_BO_FLAG_CPU_ADDR_MIRROR)
return true;
/*
* For eviction / restore on suspend / resume objects pinned in VRAM
@ -812,14 +816,14 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
}
if (ttm_bo->type == ttm_bo_type_sg) {
ret = xe_bo_move_notify(bo, ctx);
if (new_mem->mem_type == XE_PL_SYSTEM)
ret = xe_bo_move_notify(bo, ctx);
if (!ret)
ret = xe_bo_move_dmabuf(ttm_bo, new_mem);
return ret;
}
tt_has_data = ttm && (ttm_tt_is_populated(ttm) ||
(ttm->page_flags & TTM_TT_FLAG_SWAPPED));
tt_has_data = ttm && (ttm_tt_is_populated(ttm) || ttm_tt_is_swapped(ttm));
move_lacks_source = !old_mem || (handle_system_ccs ? (!bo->ccs_cleared) :
(!mem_type_is_vram(old_mem_type) && !tt_has_data));
@ -964,6 +968,20 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
dma_fence_put(fence);
xe_pm_runtime_put(xe);
/*
* CCS meta data is migrated from TT -> SMEM. So, let us detach the
* BBs from BO as it is no longer needed.
*/
if (IS_VF_CCS_BB_VALID(xe, bo) && old_mem_type == XE_PL_TT &&
new_mem->mem_type == XE_PL_SYSTEM)
xe_sriov_vf_ccs_detach_bo(bo);
if (IS_SRIOV_VF(xe) &&
((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
(old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
handle_system_ccs)
ret = xe_sriov_vf_ccs_attach_bo(bo);
out:
if ((!ttm_bo->resource || ttm_bo->resource->mem_type == XE_PL_SYSTEM) &&
ttm_bo->ttm) {
@ -974,6 +992,9 @@ out:
if (timeout < 0)
ret = timeout;
if (IS_VF_CCS_BB_VALID(xe, bo))
xe_sriov_vf_ccs_detach_bo(bo);
xe_tt_unmap_sg(xe, ttm_bo->ttm);
}
@ -1501,9 +1522,14 @@ static void xe_ttm_bo_release_notify(struct ttm_buffer_object *ttm_bo)
static void xe_ttm_bo_delete_mem_notify(struct ttm_buffer_object *ttm_bo)
{
struct xe_bo *bo = ttm_to_xe_bo(ttm_bo);
if (!xe_bo_is_xe_bo(ttm_bo))
return;
if (IS_VF_CCS_BB_VALID(ttm_to_xe_device(ttm_bo->bdev), bo))
xe_sriov_vf_ccs_detach_bo(bo);
/*
* Object is idle and about to be destroyed. Release the
* dma-buf attachment.
@ -1685,6 +1711,18 @@ static void xe_gem_object_close(struct drm_gem_object *obj,
}
}
static bool should_migrate_to_smem(struct xe_bo *bo)
{
/*
* NOTE: The following atomic checks are platform-specific. For example,
* if a device supports CXL atomics, these may not be necessary or
* may behave differently.
*/
return bo->attr.atomic_access == DRM_XE_ATOMIC_GLOBAL ||
bo->attr.atomic_access == DRM_XE_ATOMIC_CPU;
}
static vm_fault_t xe_gem_fault(struct vm_fault *vmf)
{
struct ttm_buffer_object *tbo = vmf->vma->vm_private_data;
@ -1693,7 +1731,7 @@ static vm_fault_t xe_gem_fault(struct vm_fault *vmf)
struct xe_bo *bo = ttm_to_xe_bo(tbo);
bool needs_rpm = bo->flags & XE_BO_FLAG_VRAM_MASK;
vm_fault_t ret;
int idx;
int idx, r = 0;
if (needs_rpm)
xe_pm_runtime_get(xe);
@ -1705,25 +1743,40 @@ static vm_fault_t xe_gem_fault(struct vm_fault *vmf)
if (drm_dev_enter(ddev, &idx)) {
trace_xe_bo_cpu_fault(bo);
ret = ttm_bo_vm_fault_reserved(vmf, vmf->vma->vm_page_prot,
TTM_BO_VM_NUM_PREFAULT);
if (should_migrate_to_smem(bo)) {
xe_assert(xe, bo->flags & XE_BO_FLAG_SYSTEM);
r = xe_bo_migrate(bo, XE_PL_TT);
if (r == -EBUSY || r == -ERESTARTSYS || r == -EINTR)
ret = VM_FAULT_NOPAGE;
else if (r)
ret = VM_FAULT_SIGBUS;
}
if (!ret)
ret = ttm_bo_vm_fault_reserved(vmf,
vmf->vma->vm_page_prot,
TTM_BO_VM_NUM_PREFAULT);
drm_dev_exit(idx);
if (ret == VM_FAULT_RETRY &&
!(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))
goto out;
/*
* ttm_bo_vm_reserve() already has dma_resv_lock.
*/
if (ret == VM_FAULT_NOPAGE &&
mem_type_is_vram(tbo->resource->mem_type)) {
mutex_lock(&xe->mem_access.vram_userfault.lock);
if (list_empty(&bo->vram_userfault_link))
list_add(&bo->vram_userfault_link,
&xe->mem_access.vram_userfault.list);
mutex_unlock(&xe->mem_access.vram_userfault.lock);
}
} else {
ret = ttm_bo_vm_dummy_page(vmf, vmf->vma->vm_page_prot);
}
if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))
goto out;
/*
* ttm_bo_vm_reserve() already has dma_resv_lock.
*/
if (ret == VM_FAULT_NOPAGE && mem_type_is_vram(tbo->resource->mem_type)) {
mutex_lock(&xe->mem_access.vram_userfault.lock);
if (list_empty(&bo->vram_userfault_link))
list_add(&bo->vram_userfault_link, &xe->mem_access.vram_userfault.list);
mutex_unlock(&xe->mem_access.vram_userfault.lock);
}
dma_resv_unlock(tbo->base.resv);
out:
if (needs_rpm)
@ -2438,7 +2491,6 @@ int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict)
.no_wait_gpu = false,
.gfp_retry_mayfail = true,
};
struct pin_cookie cookie;
int ret;
if (vm) {
@ -2449,10 +2501,10 @@ int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict)
ctx.resv = xe_vm_resv(vm);
}
cookie = xe_vm_set_validating(vm, allow_res_evict);
xe_vm_set_validating(vm, allow_res_evict);
trace_xe_bo_validate(bo);
ret = ttm_bo_validate(&bo->ttm, &bo->placement, &ctx);
xe_vm_clear_validating(vm, allow_res_evict, cookie);
xe_vm_clear_validating(vm, allow_res_evict);
return ret;
}

4
drivers/gpu/drm/xe/xe_bo.h
View File

@ -12,6 +12,7 @@
#include "xe_macros.h"
#include "xe_vm_types.h"
#include "xe_vm.h"
#include "xe_vram_types.h"
#define XE_DEFAULT_GTT_SIZE_MB 3072ULL /* 3GB by default */
@ -23,8 +24,9 @@
#define XE_BO_FLAG_VRAM_MASK (XE_BO_FLAG_VRAM0 | XE_BO_FLAG_VRAM1)
/* -- */
#define XE_BO_FLAG_STOLEN BIT(4)
#define XE_BO_FLAG_VRAM(vram) (XE_BO_FLAG_VRAM0 << ((vram)->id))
#define XE_BO_FLAG_VRAM_IF_DGFX(tile) (IS_DGFX(tile_to_xe(tile)) ? \
XE_BO_FLAG_VRAM0 << (tile)->id : \
XE_BO_FLAG_VRAM((tile)->mem.vram) : \
XE_BO_FLAG_SYSTEM)
#define XE_BO_FLAG_GGTT BIT(5)
#define XE_BO_FLAG_IGNORE_MIN_PAGE_SIZE BIT(6)

12
drivers/gpu/drm/xe/xe_bo_types.h
View File

@ -9,6 +9,7 @@
#include <linux/iosys-map.h>
#include <drm/drm_gpusvm.h>
#include <drm/drm_pagemap.h>
#include <drm/ttm/ttm_bo.h>
#include <drm/ttm/ttm_device.h>
#include <drm/ttm/ttm_placement.h>
@ -60,6 +61,14 @@ struct xe_bo {
*/
struct list_head client_link;
#endif
/** @attr: User controlled attributes for bo */
struct {
/**
* @atomic_access: type of atomic access bo needs
* protected by bo dma-resv lock
*/
u32 atomic_access;
} attr;
/**
* @pxp_key_instance: PXP key instance this BO was created against. A
* 0 in this variable indicates that the BO does not use PXP encryption.
@ -76,6 +85,9 @@ struct xe_bo {
/** @ccs_cleared */
bool ccs_cleared;
/** @bb_ccs_rw: BB instructions of CCS read/write. Valid only for VF */
struct xe_bb *bb_ccs[XE_SRIOV_VF_CCS_CTX_COUNT];
/**
* @cpu_caching: CPU caching mode. Currently only used for userspace
* objects. Exceptions are system memory on DGFX, which is always

346
drivers/gpu/drm/xe/xe_configfs.c
View File

@ -5,6 +5,7 @@
#include <linux/bitops.h>
#include <linux/configfs.h>
#include <linux/cleanup.h>
#include <linux/find.h>
#include <linux/init.h>
#include <linux/module.h>
@ -12,9 +13,9 @@
#include <linux/string.h>
#include "xe_configfs.h"
#include "xe_module.h"
#include "xe_hw_engine_types.h"
#include "xe_module.h"
#include "xe_pci_types.h"
/**
* DOC: Xe Configfs
@ -23,23 +24,45 @@
* =========
*
* Configfs is a filesystem-based manager of kernel objects. XE KMD registers a
* configfs subsystem called ``'xe'`` that creates a directory in the mounted configfs directory
* The user can create devices under this directory and configure them as necessary
* See Documentation/filesystems/configfs.rst for more information about how configfs works.
* configfs subsystem called ``xe`` that creates a directory in the mounted
* configfs directory. The user can create devices under this directory and
* configure them as necessary. See Documentation/filesystems/configfs.rst for
* more information about how configfs works.
*
* Create devices
* ===============
* ==============
*
* In order to create a device, the user has to create a directory inside ``'xe'``::
* To create a device, the ``xe`` module should already be loaded, but some
* attributes can only be set before binding the device. It can be accomplished
* by blocking the driver autoprobe:
*
* mkdir /sys/kernel/config/xe/0000:03:00.0/
* # echo 0 > /sys/bus/pci/drivers_autoprobe
* # modprobe xe
*
* In order to create a device, the user has to create a directory inside ``xe``::
*
* # mkdir /sys/kernel/config/xe/0000:03:00.0/
*
* Every device created is populated by the driver with entries that can be
* used to configure it::
*
* /sys/kernel/config/xe/
* .. 0000:03:00.0/
* ... survivability_mode
* ├── 0000:00:02.0
* │   └── ...
* ├── 0000:00:02.1
* │   └── ...
* :
* └── 0000:03:00.0
* ├── survivability_mode
* ├── engines_allowed
* └── enable_psmi
*
* After configuring the attributes as per next section, the device can be
* probed with::
*
* # echo 0000:03:00.0 > /sys/bus/pci/drivers/xe/bind
* # # or
* # echo 0000:03:00.0 > /sys/bus/pci/drivers_probe
*
* Configure Attributes
* ====================
@ -51,7 +74,8 @@
* effect when probing the device. Example to enable it::
*
* # echo 1 > /sys/kernel/config/xe/0000:03:00.0/survivability_mode
* # echo 0000:03:00.0 > /sys/bus/pci/drivers/xe/bind (Enters survivability mode if supported)
*
* This attribute can only be set before binding to the device.
*
* Allowed engines:
* ----------------
@ -77,24 +101,52 @@
* available for migrations, but it's disabled. This is intended for debugging
* purposes only.
*
* This attribute can only be set before binding to the device.
*
* PSMI
* ----
*
* Enable extra debugging capabilities to trace engine execution. Only useful
* during early platform enabling and requires additional hardware connected.
* Once it's enabled, additionals WAs are added and runtime configuration is
* done via debugfs. Example to enable it::
*
* # echo 1 > /sys/kernel/config/xe/0000:03:00.0/enable_psmi
*
* This attribute can only be set before binding to the device.
*
* Remove devices
* ==============
*
* The created device directories can be removed using ``rmdir``::
*
* rmdir /sys/kernel/config/xe/0000:03:00.0/
* # rmdir /sys/kernel/config/xe/0000:03:00.0/
*/
struct xe_config_device {
struct xe_config_group_device {
struct config_group group;
bool survivability_mode;
u64 engines_allowed;
struct xe_config_device {
u64 engines_allowed;
bool survivability_mode;
bool enable_psmi;
} config;
/* protects attributes */
struct mutex lock;
};
static const struct xe_config_device device_defaults = {
.engines_allowed = U64_MAX,
.survivability_mode = false,
.enable_psmi = false,
};
static void set_device_defaults(struct xe_config_device *config)
{
*config = device_defaults;
}
struct engine_info {
const char *cls;
u64 mask;
@ -113,9 +165,40 @@ static const struct engine_info engine_info[] = {
{ .cls = "gsccs", .mask = XE_HW_ENGINE_GSCCS_MASK },
};
static struct xe_config_group_device *to_xe_config_group_device(struct config_item *item)
{
return container_of(to_config_group(item), struct xe_config_group_device, group);
}
static struct xe_config_device *to_xe_config_device(struct config_item *item)
{
return container_of(to_config_group(item), struct xe_config_device, group);
return &to_xe_config_group_device(item)->config;
}
static bool is_bound(struct xe_config_group_device *dev)
{
unsigned int domain, bus, slot, function;
struct pci_dev *pdev;
const char *name;
bool ret;
lockdep_assert_held(&dev->lock);
name = dev->group.cg_item.ci_name;
if (sscanf(name, "%x:%x:%x.%x", &domain, &bus, &slot, &function) != 4)
return false;
pdev = pci_get_domain_bus_and_slot(domain, bus, PCI_DEVFN(slot, function));
if (!pdev)
return false;
ret = pci_get_drvdata(pdev);
pci_dev_put(pdev);
if (ret)
pci_dbg(pdev, "Already bound to driver\n");
return ret;
}
static ssize_t survivability_mode_show(struct config_item *item, char *page)
@ -127,7 +210,7 @@ static ssize_t survivability_mode_show(struct config_item *item, char *page)
static ssize_t survivability_mode_store(struct config_item *item, const char *page, size_t len)
{
struct xe_config_device *dev = to_xe_config_device(item);
struct xe_config_group_device *dev = to_xe_config_group_device(item);
bool survivability_mode;
int ret;
@ -135,9 +218,11 @@ static ssize_t survivability_mode_store(struct config_item *item, const char *pa
if (ret)
return ret;
mutex_lock(&dev->lock);
dev->survivability_mode = survivability_mode;
mutex_unlock(&dev->lock);
guard(mutex)(&dev->lock);
if (is_bound(dev))
return -EBUSY;
dev->config.survivability_mode = survivability_mode;
return len;
}
@ -199,7 +284,7 @@ static bool lookup_engine_mask(const char *pattern, u64 *mask)
static ssize_t engines_allowed_store(struct config_item *item, const char *page,
size_t len)
{
struct xe_config_device *dev = to_xe_config_device(item);
struct xe_config_group_device *dev = to_xe_config_group_device(item);
size_t patternlen, p;
u64 mask, val = 0;
@ -219,25 +304,55 @@ static ssize_t engines_allowed_store(struct config_item *item, const char *page,
val |= mask;
}
mutex_lock(&dev->lock);
dev->engines_allowed = val;
mutex_unlock(&dev->lock);
guard(mutex)(&dev->lock);
if (is_bound(dev))
return -EBUSY;
dev->config.engines_allowed = val;
return len;
}
CONFIGFS_ATTR(, survivability_mode);
static ssize_t enable_psmi_show(struct config_item *item, char *page)
{
struct xe_config_device *dev = to_xe_config_device(item);
return sprintf(page, "%d\n", dev->enable_psmi);
}
static ssize_t enable_psmi_store(struct config_item *item, const char *page, size_t len)
{
struct xe_config_group_device *dev = to_xe_config_group_device(item);
bool val;
int ret;
ret = kstrtobool(page, &val);
if (ret)
return ret;
guard(mutex)(&dev->lock);
if (is_bound(dev))
return -EBUSY;
dev->config.enable_psmi = val;
return len;
}
CONFIGFS_ATTR(, enable_psmi);
CONFIGFS_ATTR(, engines_allowed);
CONFIGFS_ATTR(, survivability_mode);
static struct configfs_attribute *xe_config_device_attrs[] = {
&attr_survivability_mode,
&attr_enable_psmi,
&attr_engines_allowed,
&attr_survivability_mode,
NULL,
};
static void xe_config_device_release(struct config_item *item)
{
struct xe_config_device *dev = to_xe_config_device(item);
struct xe_config_group_device *dev = to_xe_config_group_device(item);
mutex_destroy(&dev->lock);
kfree(dev);
@ -253,29 +368,81 @@ static const struct config_item_type xe_config_device_type = {
.ct_owner = THIS_MODULE,
};
static const struct xe_device_desc *xe_match_desc(struct pci_dev *pdev)
{
struct device_driver *driver = driver_find("xe", &pci_bus_type);
struct pci_driver *drv = to_pci_driver(driver);
const struct pci_device_id *ids = drv ? drv->id_table : NULL;
const struct pci_device_id *found = pci_match_id(ids, pdev);
return found ? (const void *)found->driver_data : NULL;
}
static struct pci_dev *get_physfn_instead(struct pci_dev *virtfn)
{
struct pci_dev *physfn = pci_physfn(virtfn);
pci_dev_get(physfn);
pci_dev_put(virtfn);
return physfn;
}
static struct config_group *xe_config_make_device_group(struct config_group *group,
const char *name)
{
unsigned int domain, bus, slot, function;
struct xe_config_device *dev;
struct xe_config_group_device *dev;
const struct xe_device_desc *match;
struct pci_dev *pdev;
char canonical[16];
int vfnumber = 0;
int ret;
ret = sscanf(name, "%04x:%02x:%02x.%x", &domain, &bus, &slot, &function);
ret = sscanf(name, "%x:%x:%x.%x", &domain, &bus, &slot, &function);
if (ret != 4)
return ERR_PTR(-EINVAL);
ret = scnprintf(canonical, sizeof(canonical), "%04x:%02x:%02x.%d", domain, bus,
PCI_SLOT(PCI_DEVFN(slot, function)),
PCI_FUNC(PCI_DEVFN(slot, function)));
if (ret != 12 || strcmp(name, canonical))
return ERR_PTR(-EINVAL);
pdev = pci_get_domain_bus_and_slot(domain, bus, PCI_DEVFN(slot, function));
if (!pdev && function)
pdev = pci_get_domain_bus_and_slot(domain, bus, PCI_DEVFN(slot, 0));
if (!pdev && slot)
pdev = pci_get_domain_bus_and_slot(domain, bus, PCI_DEVFN(0, 0));
if (!pdev)
return ERR_PTR(-ENODEV);
if (PCI_DEVFN(slot, function) != pdev->devfn) {
pdev = get_physfn_instead(pdev);
vfnumber = PCI_DEVFN(slot, function) - pdev->devfn;
if (!dev_is_pf(&pdev->dev) || vfnumber > pci_sriov_get_totalvfs(pdev)) {
pci_dev_put(pdev);
return ERR_PTR(-ENODEV);
}
}
match = xe_match_desc(pdev);
if (match && vfnumber && !match->has_sriov) {
pci_info(pdev, "xe driver does not support VFs on this device\n");
match = NULL;
} else if (!match) {
pci_info(pdev, "xe driver does not support configuration of this device\n");
}
pci_dev_put(pdev);
if (!match)
return ERR_PTR(-ENOENT);
dev = kzalloc(sizeof(*dev), GFP_KERNEL);
if (!dev)
return ERR_PTR(-ENOMEM);
/* Default values */
dev->engines_allowed = U64_MAX;
set_device_defaults(&dev->config);
config_group_init_type_name(&dev->group, name, &xe_config_device_type);
@ -302,102 +469,145 @@ static struct configfs_subsystem xe_configfs = {
},
};
static struct xe_config_device *configfs_find_group(struct pci_dev *pdev)
static struct xe_config_group_device *find_xe_config_group_device(struct pci_dev *pdev)
{
struct config_item *item;
char name[64];
snprintf(name, sizeof(name), "%04x:%02x:%02x.%x", pci_domain_nr(pdev->bus),
pdev->bus->number, PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn));
mutex_lock(&xe_configfs.su_mutex);
item = config_group_find_item(&xe_configfs.su_group, name);
item = config_group_find_item(&xe_configfs.su_group, pci_name(pdev));
mutex_unlock(&xe_configfs.su_mutex);
if (!item)
return NULL;
return to_xe_config_device(item);
return to_xe_config_group_device(item);
}
static void dump_custom_dev_config(struct pci_dev *pdev,
struct xe_config_group_device *dev)
{
#define PRI_CUSTOM_ATTR(fmt_, attr_) do { \
if (dev->config.attr_ != device_defaults.attr_) \
pci_info(pdev, "configfs: " __stringify(attr_) " = " fmt_ "\n", \
dev->config.attr_); \
} while (0)
PRI_CUSTOM_ATTR("%llx", engines_allowed);
PRI_CUSTOM_ATTR("%d", enable_psmi);
PRI_CUSTOM_ATTR("%d", survivability_mode);
#undef PRI_CUSTOM_ATTR
}
/**
* xe_configfs_check_device() - Test if device was configured by configfs
* @pdev: the &pci_dev device to test
*
* Try to find the configfs group that belongs to the specified pci device
* and print a diagnostic message if different than the default value.
*/
void xe_configfs_check_device(struct pci_dev *pdev)
{
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
if (!dev)
return;
/* memcmp here is safe as both are zero-initialized */
if (memcmp(&dev->config, &device_defaults, sizeof(dev->config))) {
pci_info(pdev, "Found custom settings in configfs\n");
dump_custom_dev_config(pdev, dev);
}
config_group_put(&dev->group);
}
/**
* xe_configfs_get_survivability_mode - get configfs survivability mode attribute
* @pdev: pci device
*
* find the configfs group that belongs to the pci device and return
* the survivability mode attribute
*
* Return: survivability mode if config group is found, false otherwise
* Return: survivability_mode attribute in configfs
*/
bool xe_configfs_get_survivability_mode(struct pci_dev *pdev)
{
struct xe_config_device *dev = configfs_find_group(pdev);
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
bool mode;
if (!dev)
return false;
return device_defaults.survivability_mode;
mode = dev->survivability_mode;
config_item_put(&dev->group.cg_item);
mode = dev->config.survivability_mode;
config_group_put(&dev->group);
return mode;
}
/**
* xe_configfs_clear_survivability_mode - clear configfs survivability mode attribute
* xe_configfs_clear_survivability_mode - clear configfs survivability mode
* @pdev: pci device
*
* find the configfs group that belongs to the pci device and clear survivability
* mode attribute
*/
void xe_configfs_clear_survivability_mode(struct pci_dev *pdev)
{
struct xe_config_device *dev = configfs_find_group(pdev);
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
if (!dev)
return;
mutex_lock(&dev->lock);
dev->survivability_mode = 0;
mutex_unlock(&dev->lock);
guard(mutex)(&dev->lock);
dev->config.survivability_mode = 0;
config_item_put(&dev->group.cg_item);
config_group_put(&dev->group);
}
/**
* xe_configfs_get_engines_allowed - get engine allowed mask from configfs
* @pdev: pci device
*
* Find the configfs group that belongs to the pci device and return
* the mask of engines allowed to be used.
*
* Return: engine mask with allowed engines
* Return: engine mask with allowed engines set in configfs
*/
u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev)
{
struct xe_config_device *dev = configfs_find_group(pdev);
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
u64 engines_allowed;
if (!dev)
return U64_MAX;
return device_defaults.engines_allowed;
engines_allowed = dev->engines_allowed;
config_item_put(&dev->group.cg_item);
engines_allowed = dev->config.engines_allowed;
config_group_put(&dev->group);
return engines_allowed;
}
/**
* xe_configfs_get_psmi_enabled - get configfs enable_psmi setting
* @pdev: pci device
*
* Return: enable_psmi setting in configfs
*/
bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev)
{
struct xe_config_group_device *dev = find_xe_config_group_device(pdev);
bool ret;
if (!dev)
return false;
ret = dev->config.enable_psmi;
config_item_put(&dev->group.cg_item);
return ret;
}
int __init xe_configfs_init(void)
{
struct config_group *root = &xe_configfs.su_group;
int ret;
config_group_init(root);
config_group_init(&xe_configfs.su_group);
mutex_init(&xe_configfs.su_mutex);
ret = configfs_register_subsystem(&xe_configfs);
if (ret) {
pr_err("Error %d while registering %s subsystem\n",
ret, root->cg_item.ci_namebuf);
mutex_destroy(&xe_configfs.su_mutex);
return ret;
}
@ -407,5 +617,5 @@ int __init xe_configfs_init(void)
void __exit xe_configfs_exit(void)
{
configfs_unregister_subsystem(&xe_configfs);
mutex_destroy(&xe_configfs.su_mutex);
}

4
drivers/gpu/drm/xe/xe_configfs.h
View File

@ -13,15 +13,19 @@ struct pci_dev;
#if IS_ENABLED(CONFIG_CONFIGFS_FS)
int xe_configfs_init(void);
void xe_configfs_exit(void);
void xe_configfs_check_device(struct pci_dev *pdev);
bool xe_configfs_get_survivability_mode(struct pci_dev *pdev);
void xe_configfs_clear_survivability_mode(struct pci_dev *pdev);
u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev);
bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev);
#else
static inline int xe_configfs_init(void) { return 0; }
static inline void xe_configfs_exit(void) { }
static inline void xe_configfs_check_device(struct pci_dev *pdev) { }
static inline bool xe_configfs_get_survivability_mode(struct pci_dev *pdev) { return false; }
static inline void xe_configfs_clear_survivability_mode(struct pci_dev *pdev) { }
static inline u64 xe_configfs_get_engines_allowed(struct pci_dev *pdev) { return U64_MAX; }
static inline bool xe_configfs_get_psmi_enabled(struct pci_dev *pdev) { return false; }
#endif
#endif

114
drivers/gpu/drm/xe/xe_debugfs.c
View File

@ -11,18 +11,22 @@
#include <drm/drm_debugfs.h>
#include "regs/xe_pmt.h"
#include "xe_bo.h"
#include "xe_device.h"
#include "xe_force_wake.h"
#include "xe_gt_debugfs.h"
#include "xe_gt_printk.h"
#include "xe_guc_ads.h"
#include "xe_mmio.h"
#include "xe_pm.h"
#include "xe_psmi.h"
#include "xe_pxp_debugfs.h"
#include "xe_sriov.h"
#include "xe_sriov_pf.h"
#include "xe_step.h"
#include "xe_wa.h"
#include "xe_vsec.h"
#ifdef CONFIG_DRM_XE_DEBUG
#include "xe_bo_evict.h"
@ -31,6 +35,24 @@
#endif
DECLARE_FAULT_ATTR(gt_reset_failure);
DECLARE_FAULT_ATTR(inject_csc_hw_error);
static void read_residency_counter(struct xe_device *xe, struct xe_mmio *mmio,
u32 offset, char *name, struct drm_printer *p)
{
u64 residency = 0;
int ret;
ret = xe_pmt_telem_read(to_pci_dev(xe->drm.dev),
xe_mmio_read32(mmio, PUNIT_TELEMETRY_GUID),
&residency, offset, sizeof(residency));
if (ret != sizeof(residency)) {
drm_warn(&xe->drm, "%s counter failed to read, ret %d\n", name, ret);
return;
}
drm_printf(p, "%s : %llu\n", name, residency);
}
static struct xe_device *node_to_xe(struct drm_info_node *node)
{
@ -102,12 +124,72 @@ static int workaround_info(struct seq_file *m, void *data)
return 0;
}
static int dgfx_pkg_residencies_show(struct seq_file *m, void *data)
{
struct xe_device *xe;
struct xe_mmio *mmio;
struct drm_printer p;
xe = node_to_xe(m->private);
p = drm_seq_file_printer(m);
xe_pm_runtime_get(xe);
mmio = xe_root_tile_mmio(xe);
struct {
u32 offset;
char *name;
} residencies[] = {
{BMG_G2_RESIDENCY_OFFSET, "Package G2"},
{BMG_G6_RESIDENCY_OFFSET, "Package G6"},
{BMG_G8_RESIDENCY_OFFSET, "Package G8"},
{BMG_G10_RESIDENCY_OFFSET, "Package G10"},
{BMG_MODS_RESIDENCY_OFFSET, "Package ModS"}
};
for (int i = 0; i < ARRAY_SIZE(residencies); i++)
read_residency_counter(xe, mmio, residencies[i].offset, residencies[i].name, &p);
xe_pm_runtime_put(xe);
return 0;
}
static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
{
struct xe_device *xe;
struct xe_mmio *mmio;
struct drm_printer p;
xe = node_to_xe(m->private);
p = drm_seq_file_printer(m);
xe_pm_runtime_get(xe);
mmio = xe_root_tile_mmio(xe);
struct {
u32 offset;
char *name;
} residencies[] = {
{BMG_PCIE_LINK_L0_RESIDENCY_OFFSET, "PCIE LINK L0 RESIDENCY"},
{BMG_PCIE_LINK_L1_RESIDENCY_OFFSET, "PCIE LINK L1 RESIDENCY"},
{BMG_PCIE_LINK_L1_2_RESIDENCY_OFFSET, "PCIE LINK L1.2 RESIDENCY"}
};
for (int i = 0; i < ARRAY_SIZE(residencies); i++)
read_residency_counter(xe, mmio, residencies[i].offset, residencies[i].name, &p);
xe_pm_runtime_put(xe);
return 0;
}
static const struct drm_info_list debugfs_list[] = {
{"info", info, 0},
{ .name = "sriov_info", .show = sriov_info, },
{ .name = "workarounds", .show = workaround_info, },
};
static const struct drm_info_list debugfs_residencies[] = {
{ .name = "dgfx_pkg_residencies", .show = dgfx_pkg_residencies_show, },
{ .name = "dgfx_pcie_link_residencies", .show = dgfx_pcie_link_residencies_show, },
};
static int forcewake_open(struct inode *inode, struct file *file)
{
struct xe_device *xe = inode->i_private;
@ -247,20 +329,47 @@ static const struct file_operations atomic_svm_timeslice_ms_fops = {
.write = atomic_svm_timeslice_ms_set,
};
static void create_tile_debugfs(struct xe_tile *tile, struct dentry *root)
{
char name[8];
snprintf(name, sizeof(name), "tile%u", tile->id);
tile->debugfs = debugfs_create_dir(name, root);
if (IS_ERR(tile->debugfs))
return;
/*
* Store the xe_tile pointer as private data of the tile/ directory
* node so other tile specific attributes under that directory may
* refer to it by looking at its parent node private data.
*/
tile->debugfs->d_inode->i_private = tile;
}
void xe_debugfs_register(struct xe_device *xe)
{
struct ttm_device *bdev = &xe->ttm;
struct drm_minor *minor = xe->drm.primary;
struct dentry *root = minor->debugfs_root;
struct ttm_resource_manager *man;
struct xe_tile *tile;
struct xe_gt *gt;
u32 mem_type;
u8 tile_id;
u8 id;
drm_debugfs_create_files(debugfs_list,
ARRAY_SIZE(debugfs_list),
root, minor);
if (xe->info.platform == XE_BATTLEMAGE) {
drm_debugfs_create_files(debugfs_residencies,
ARRAY_SIZE(debugfs_residencies),
root, minor);
fault_create_debugfs_attr("inject_csc_hw_error", root,
&inject_csc_hw_error);
}
debugfs_create_file("forcewake_all", 0400, root, xe,
&forcewake_all_fops);
@ -288,11 +397,16 @@ void xe_debugfs_register(struct xe_device *xe)
if (man)
ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
for_each_tile(tile, xe, tile_id)
create_tile_debugfs(tile, root);
for_each_gt(gt, xe, id)
xe_gt_debugfs_register(gt);
xe_pxp_debugfs_register(xe->pxp);
xe_psmi_debugfs_register(xe);
fault_create_debugfs_attr("fail_gt_reset", root, &gt_reset_failure);
if (IS_SRIOV_PF(xe))

29
drivers/gpu/drm/xe/xe_dep_job_types.h Normal file
View File

@ -0,0 +1,29 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef _XE_DEP_JOB_TYPES_H_
#define _XE_DEP_JOB_TYPES_H_
#include <drm/gpu_scheduler.h>
struct xe_dep_job;
/** struct xe_dep_job_ops - Generic Xe dependency job operations */
struct xe_dep_job_ops {
/** @run_job: Run generic Xe dependency job */
struct dma_fence *(*run_job)(struct xe_dep_job *job);
/** @free_job: Free generic Xe dependency job */
void (*free_job)(struct xe_dep_job *job);
};
/** struct xe_dep_job - Generic dependency Xe job */
struct xe_dep_job {
/** @drm: base DRM scheduler job */
struct drm_sched_job drm;
/** @ops: dependency job operations */
const struct xe_dep_job_ops *ops;
};
#endif

143
drivers/gpu/drm/xe/xe_dep_scheduler.c Normal file
View File

@ -0,0 +1,143 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2025 Intel Corporation
*/
#include <linux/slab.h>
#include <drm/gpu_scheduler.h>
#include "xe_dep_job_types.h"
#include "xe_dep_scheduler.h"
#include "xe_device_types.h"
/**
* DOC: Xe Dependency Scheduler
*
* The Xe dependency scheduler is a simple wrapper built around the DRM
* scheduler to execute jobs once their dependencies are resolved (i.e., all
* input fences specified as dependencies are signaled). The jobs that are
* executed contain virtual functions to run (execute) and free the job,
* allowing a single dependency scheduler to handle jobs performing different
* operations.
*
* Example use cases include deferred resource freeing, TLB invalidations after
* bind jobs, etc.
*/
/** struct xe_dep_scheduler - Generic Xe dependency scheduler */
struct xe_dep_scheduler {
/** @sched: DRM GPU scheduler */
struct drm_gpu_scheduler sched;
/** @entity: DRM scheduler entity */
struct drm_sched_entity entity;
/** @rcu: For safe freeing of exported dma fences */
struct rcu_head rcu;
};
static struct dma_fence *xe_dep_scheduler_run_job(struct drm_sched_job *drm_job)
{
struct xe_dep_job *dep_job =
container_of(drm_job, typeof(*dep_job), drm);
return dep_job->ops->run_job(dep_job);
}
static void xe_dep_scheduler_free_job(struct drm_sched_job *drm_job)
{
struct xe_dep_job *dep_job =
container_of(drm_job, typeof(*dep_job), drm);
dep_job->ops->free_job(dep_job);
}
static const struct drm_sched_backend_ops sched_ops = {
.run_job = xe_dep_scheduler_run_job,
.free_job = xe_dep_scheduler_free_job,
};
/**
* xe_dep_scheduler_create() - Generic Xe dependency scheduler create
* @xe: Xe device
* @submit_wq: Submit workqueue struct (can be NULL)
* @name: Name of dependency scheduler
* @job_limit: Max dependency jobs that can be scheduled
*
* Create a generic Xe dependency scheduler and initialize internal DRM
* scheduler objects.
*
* Return: Generic Xe dependency scheduler object on success, ERR_PTR failure
*/
struct xe_dep_scheduler *
xe_dep_scheduler_create(struct xe_device *xe,
struct workqueue_struct *submit_wq,
const char *name, u32 job_limit)
{
struct xe_dep_scheduler *dep_scheduler;
struct drm_gpu_scheduler *sched;
const struct drm_sched_init_args args = {
.ops = &sched_ops,
.submit_wq = submit_wq,
.num_rqs = 1,
.credit_limit = job_limit,
.timeout = MAX_SCHEDULE_TIMEOUT,
.name = name,
.dev = xe->drm.dev,
};
int err;
dep_scheduler = kzalloc(sizeof(*dep_scheduler), GFP_KERNEL);
if (!dep_scheduler)
return ERR_PTR(-ENOMEM);
err = drm_sched_init(&dep_scheduler->sched, &args);
if (err)
goto err_free;
sched = &dep_scheduler->sched;
err = drm_sched_entity_init(&dep_scheduler->entity, 0, &sched, 1, NULL);
if (err)
goto err_sched;
init_rcu_head(&dep_scheduler->rcu);
return dep_scheduler;
err_sched:
drm_sched_fini(&dep_scheduler->sched);
err_free:
kfree(dep_scheduler);
return ERR_PTR(err);
}
/**
* xe_dep_scheduler_fini() - Generic Xe dependency scheduler finalize
* @dep_scheduler: Generic Xe dependency scheduler object
*
* Finalize internal DRM scheduler objects and free generic Xe dependency
* scheduler object
*/
void xe_dep_scheduler_fini(struct xe_dep_scheduler *dep_scheduler)
{
drm_sched_entity_fini(&dep_scheduler->entity);
drm_sched_fini(&dep_scheduler->sched);
/*
* RCU free due sched being exported via DRM scheduler fences
* (timeline name).
*/
kfree_rcu(dep_scheduler, rcu);
}
/**
* xe_dep_scheduler_entity() - Retrieve a generic Xe dependency scheduler
* DRM scheduler entity
* @dep_scheduler: Generic Xe dependency scheduler object
*
* Return: The generic Xe dependency scheduler's DRM scheduler entity
*/
struct drm_sched_entity *
xe_dep_scheduler_entity(struct xe_dep_scheduler *dep_scheduler)
{
return &dep_scheduler->entity;
}

21
drivers/gpu/drm/xe/xe_dep_scheduler.h Normal file
View File

@ -0,0 +1,21 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#include <linux/types.h>
struct drm_sched_entity;
struct workqueue_struct;
struct xe_dep_scheduler;
struct xe_device;
struct xe_dep_scheduler *
xe_dep_scheduler_create(struct xe_device *xe,
struct workqueue_struct *submit_wq,
const char *name, u32 job_limit);
void xe_dep_scheduler_fini(struct xe_dep_scheduler *dep_scheduler);
struct drm_sched_entity *
xe_dep_scheduler_entity(struct xe_dep_scheduler *dep_scheduler);

112
drivers/gpu/drm/xe/xe_device.c
View File

@ -54,6 +54,7 @@
#include "xe_pcode.h"
#include "xe_pm.h"
#include "xe_pmu.h"
#include "xe_psmi.h"
#include "xe_pxp.h"
#include "xe_query.h"
#include "xe_shrinker.h"
@ -63,7 +64,9 @@
#include "xe_ttm_stolen_mgr.h"
#include "xe_ttm_sys_mgr.h"
#include "xe_vm.h"
#include "xe_vm_madvise.h"
#include "xe_vram.h"
#include "xe_vram_types.h"
#include "xe_vsec.h"
#include "xe_wait_user_fence.h"
#include "xe_wa.h"
@ -200,6 +203,9 @@ static const struct drm_ioctl_desc xe_ioctls[] = {
DRM_IOCTL_DEF_DRV(XE_WAIT_USER_FENCE, xe_wait_user_fence_ioctl,
DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(XE_OBSERVATION, xe_observation_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(XE_MADVISE, xe_vm_madvise_ioctl, DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(XE_VM_QUERY_MEM_RANGE_ATTRS, xe_vm_query_vmas_attrs_ioctl,
DRM_RENDER_ALLOW),
};
static long xe_drm_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
@ -688,6 +694,21 @@ static void sriov_update_device_info(struct xe_device *xe)
}
}
static int xe_device_vram_alloc(struct xe_device *xe)
{
struct xe_vram_region *vram;
if (!IS_DGFX(xe))
return 0;
vram = drmm_kzalloc(&xe->drm, sizeof(*vram), GFP_KERNEL);
if (!vram)
return -ENOMEM;
xe->mem.vram = vram;
return 0;
}
/**
* xe_device_probe_early: Device early probe
* @xe: xe device instance
@ -722,7 +743,7 @@ int xe_device_probe_early(struct xe_device *xe)
* possible, but still return the previous error for error
* propagation
*/
err = xe_survivability_mode_enable(xe);
err = xe_survivability_mode_boot_enable(xe);
if (err)
return err;
@ -735,6 +756,10 @@ int xe_device_probe_early(struct xe_device *xe)
xe->wedged.mode = xe_modparam.wedged_mode;
err = xe_device_vram_alloc(xe);
if (err)
return err;
return 0;
}
ALLOW_ERROR_INJECTION(xe_device_probe_early, ERRNO); /* See xe_pci_probe() */
@ -863,7 +888,7 @@ int xe_device_probe(struct xe_device *xe)
}
if (xe->tiles->media_gt &&
XE_WA(xe->tiles->media_gt, 15015404425_disable))
XE_GT_WA(xe->tiles->media_gt, 15015404425_disable))
XE_DEVICE_WA_DISABLE(xe, 15015404425);
err = xe_devcoredump_init(xe);
@ -888,6 +913,10 @@ int xe_device_probe(struct xe_device *xe)
if (err)
return err;
err = xe_psmi_init(xe);
if (err)
return err;
err = drm_dev_register(&xe->drm, 0);
if (err)
return err;
@ -921,6 +950,10 @@ int xe_device_probe(struct xe_device *xe)
xe_vsec_init(xe);
err = xe_sriov_late_init(xe);
if (err)
goto err_unregister_display;
return devm_add_action_or_reset(xe->drm.dev, xe_device_sanitize, xe);
err_unregister_display:
@ -1019,7 +1052,7 @@ void xe_device_l2_flush(struct xe_device *xe)
gt = xe_root_mmio_gt(xe);
if (!XE_WA(gt, 16023588340))
if (!XE_GT_WA(gt, 16023588340))
return;
fw_ref = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
@ -1063,7 +1096,7 @@ void xe_device_td_flush(struct xe_device *xe)
return;
root_gt = xe_root_mmio_gt(xe);
if (XE_WA(root_gt, 16023588340)) {
if (XE_GT_WA(root_gt, 16023588340)) {
/* A transient flush is not sufficient: flush the L2 */
xe_device_l2_flush(xe);
} else {
@ -1133,12 +1166,64 @@ static void xe_device_wedged_fini(struct drm_device *drm, void *arg)
xe_pm_runtime_put(xe);
}
/**
* DOC: Xe Device Wedging
*
* Xe driver uses drm device wedged uevent as documented in Documentation/gpu/drm-uapi.rst.
* When device is in wedged state, every IOCTL will be blocked and GT cannot be
* used. Certain critical errors like gt reset failure, firmware failures can cause
* the device to be wedged. The default recovery method for a wedged state
* is rebind/bus-reset.
*
* Another recovery method is vendor-specific. Below are the cases that send
* ``WEDGED=vendor-specific`` recovery method in drm device wedged uevent.
*
* Case: Firmware Flash
* --------------------
*
* Identification Hint
* +++++++++++++++++++
*
* ``WEDGED=vendor-specific`` drm device wedged uevent with
* :ref:`Runtime Survivability mode <xe-survivability-mode>` is used to notify
* admin/userspace consumer about the need for a firmware flash.
*
* Recovery Procedure
* ++++++++++++++++++
*
* Once ``WEDGED=vendor-specific`` drm device wedged uevent is received, follow
* the below steps
*
* - Check Runtime Survivability mode sysfs.
* If enabled, firmware flash is required to recover the device.
*
* /sys/bus/pci/devices/<device>/survivability_mode
*
* - Admin/userpsace consumer can use firmware flashing tools like fwupd to flash
* firmware and restore device to normal operation.
*/
/**
* xe_device_set_wedged_method - Set wedged recovery method
* @xe: xe device instance
* @method: recovery method to set
*
* Set wedged recovery method to be sent in drm wedged uevent.
*/
void xe_device_set_wedged_method(struct xe_device *xe, unsigned long method)
{
xe->wedged.method = method;
}
/**
* xe_device_declare_wedged - Declare device wedged
* @xe: xe device instance
*
* This is a final state that can only be cleared with a module
* re-probe (unbind + bind).
* This is a final state that can only be cleared with the recovery method
* specified in the drm wedged uevent. The method can be set using
* xe_device_set_wedged_method before declaring the device as wedged. If no method
* is set, reprobe (unbind/re-bind) will be sent by default.
*
* In this state every IOCTL will be blocked so the GT cannot be used.
* In general it will be called upon any critical error such as gt reset
* failure or guc loading failure. Userspace will be notified of this state
@ -1172,13 +1257,18 @@ void xe_device_declare_wedged(struct xe_device *xe)
"IOCTLs and executions are blocked. Only a rebind may clear the failure\n"
"Please file a _new_ bug report at https://gitlab.freedesktop.org/drm/xe/kernel/issues/new\n",
dev_name(xe->drm.dev));
/* Notify userspace of wedged device */
drm_dev_wedged_event(&xe->drm,
DRM_WEDGE_RECOVERY_REBIND | DRM_WEDGE_RECOVERY_BUS_RESET,
NULL);
}
for_each_gt(gt, xe, id)
xe_gt_declare_wedged(gt);
if (xe_device_wedged(xe)) {
/* If no wedge recovery method is set, use default */
if (!xe->wedged.method)
xe_device_set_wedged_method(xe, DRM_WEDGE_RECOVERY_REBIND |
DRM_WEDGE_RECOVERY_BUS_RESET);
/* Notify userspace of wedged device */
drm_dev_wedged_event(&xe->drm, xe->wedged.method, NULL);
}
}

1
drivers/gpu/drm/xe/xe_device.h
View File

@ -187,6 +187,7 @@ static inline bool xe_device_wedged(struct xe_device *xe)
return atomic_read(&xe->wedged.flag);
}
void xe_device_set_wedged_method(struct xe_device *xe, unsigned long method);
void xe_device_declare_wedged(struct xe_device *xe);
struct xe_file *xe_file_get(struct xe_file *xef);

8
drivers/gpu/drm/xe/xe_device_sysfs.c
View File

@ -76,7 +76,7 @@ lb_fan_control_version_show(struct device *dev, struct device_attribute *attr, c
{
struct xe_device *xe = pdev_to_xe_device(to_pci_dev(dev));
struct xe_tile *root = xe_device_get_root_tile(xe);
u32 cap, ver_low = FAN_TABLE, ver_high = FAN_TABLE;
u32 cap = 0, ver_low = FAN_TABLE, ver_high = FAN_TABLE;
u16 major = 0, minor = 0, hotfix = 0, build = 0;
int ret;
@ -115,7 +115,7 @@ lb_voltage_regulator_version_show(struct device *dev, struct device_attribute *a
{
struct xe_device *xe = pdev_to_xe_device(to_pci_dev(dev));
struct xe_tile *root = xe_device_get_root_tile(xe);
u32 cap, ver_low = VR_CONFIG, ver_high = VR_CONFIG;
u32 cap = 0, ver_low = VR_CONFIG, ver_high = VR_CONFIG;
u16 major = 0, minor = 0, hotfix = 0, build = 0;
int ret;
@ -153,7 +153,7 @@ static int late_bind_create_files(struct device *dev)
{
struct xe_device *xe = pdev_to_xe_device(to_pci_dev(dev));
struct xe_tile *root = xe_device_get_root_tile(xe);
u32 cap;
u32 cap = 0;
int ret;
xe_pm_runtime_get(xe);
@ -186,7 +186,7 @@ static void late_bind_remove_files(struct device *dev)
{
struct xe_device *xe = pdev_to_xe_device(to_pci_dev(dev));
struct xe_tile *root = xe_device_get_root_tile(xe);
u32 cap;
u32 cap = 0;
int ret;
xe_pm_runtime_get(xe);

86
drivers/gpu/drm/xe/xe_device_types.h
View File

@ -10,7 +10,6 @@
#include <drm/drm_device.h>
#include <drm/drm_file.h>
#include <drm/drm_pagemap.h>
#include <drm/ttm/ttm_device.h>
#include "xe_devcoredump_types.h"
@ -24,9 +23,9 @@
#include "xe_sriov_pf_types.h"
#include "xe_sriov_types.h"
#include "xe_sriov_vf_types.h"
#include "xe_sriov_vf_ccs_types.h"
#include "xe_step_types.h"
#include "xe_survivability_mode_types.h"
#include "xe_ttm_vram_mgr_types.h"
#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
#define TEST_VM_OPS_ERROR
@ -39,6 +38,7 @@ struct xe_ggtt;
struct xe_i2c;
struct xe_pat_ops;
struct xe_pxp;
struct xe_vram_region;
#define XE_BO_INVALID_OFFSET LONG_MAX
@ -71,61 +71,6 @@ struct xe_pxp;
const struct xe_tile * : (const struct xe_device *)((tile__)->xe), \
struct xe_tile * : (tile__)->xe)
/**
* struct xe_vram_region - memory region structure
* This is used to describe a memory region in xe
* device, such as HBM memory or CXL extension memory.
*/
struct xe_vram_region {
/** @io_start: IO start address of this VRAM instance */
resource_size_t io_start;
/**
* @io_size: IO size of this VRAM instance
*
* This represents how much of this VRAM we can access
* via the CPU through the VRAM BAR. This can be smaller
* than @usable_size, in which case only part of VRAM is CPU
* accessible (typically the first 256M). This
* configuration is known as small-bar.
*/
resource_size_t io_size;
/** @dpa_base: This memory regions's DPA (device physical address) base */
resource_size_t dpa_base;
/**
* @usable_size: usable size of VRAM
*
* Usable size of VRAM excluding reserved portions
* (e.g stolen mem)
*/
resource_size_t usable_size;
/**
* @actual_physical_size: Actual VRAM size
*
* Actual VRAM size including reserved portions
* (e.g stolen mem)
*/
resource_size_t actual_physical_size;
/** @mapping: pointer to VRAM mappable space */
void __iomem *mapping;
/** @ttm: VRAM TTM manager */
struct xe_ttm_vram_mgr ttm;
#if IS_ENABLED(CONFIG_DRM_XE_PAGEMAP)
/** @pagemap: Used to remap device memory as ZONE_DEVICE */
struct dev_pagemap pagemap;
/**
* @dpagemap: The struct drm_pagemap of the ZONE_DEVICE memory
* pages of this tile.
*/
struct drm_pagemap dpagemap;
/**
* @hpa_base: base host physical address
*
* This is generated when remap device memory as ZONE_DEVICE
*/
resource_size_t hpa_base;
#endif
};
/**
* struct xe_mmio - register mmio structure
*
@ -216,7 +161,7 @@ struct xe_tile {
* Although VRAM is associated with a specific tile, it can
* still be accessed by all tiles' GTs.
*/
struct xe_vram_region vram;
struct xe_vram_region *vram;
/** @mem.ggtt: Global graphics translation table */
struct xe_ggtt *ggtt;
@ -238,12 +183,18 @@ struct xe_tile {
struct {
/** @sriov.vf.ggtt_balloon: GGTT regions excluded from use. */
struct xe_ggtt_node *ggtt_balloon[2];
/** @sriov.vf.ccs: CCS read and write contexts for VF. */
struct xe_tile_vf_ccs ccs[XE_SRIOV_VF_CCS_CTX_COUNT];
} vf;
} sriov;
/** @memirq: Memory Based Interrupts. */
struct xe_memirq memirq;
/** @csc_hw_error_work: worker to report CSC HW errors */
struct work_struct csc_hw_error_work;
/** @pcode: tile's PCODE */
struct {
/** @pcode.lock: protecting tile's PCODE mailbox data */
@ -255,6 +206,9 @@ struct xe_tile {
/** @sysfs: sysfs' kobj used by xe_tile_sysfs */
struct kobject *sysfs;
/** @debugfs: debugfs directory associated with this tile */
struct dentry *debugfs;
};
/**
@ -336,8 +290,8 @@ struct xe_device {
u8 has_mbx_power_limits:1;
/** @info.has_pxp: Device has PXP support */
u8 has_pxp:1;
/** @info.has_range_tlb_invalidation: Has range based TLB invalidations */
u8 has_range_tlb_invalidation:1;
/** @info.has_range_tlb_inval: Has range based TLB invalidations */
u8 has_range_tlb_inval:1;
/** @info.has_sriov: Supports SR-IOV */
u8 has_sriov:1;
/** @info.has_usm: Device has unified shared memory support */
@ -412,7 +366,7 @@ struct xe_device {
/** @mem: memory info for device */
struct {
/** @mem.vram: VRAM info for device */
struct xe_vram_region vram;
struct xe_vram_region *vram;
/** @mem.sys_mgr: system TTM manager */
struct ttm_resource_manager sys_mgr;
/** @mem.sys_mgr: system memory shrinker. */
@ -590,6 +544,8 @@ struct xe_device {
atomic_t flag;
/** @wedged.mode: Mode controlled by kernel parameter and debugfs */
int mode;
/** @wedged.method: Recovery method to be sent in the drm device wedged uevent */
unsigned long method;
} wedged;
/** @bo_device: Struct to control async free of BOs */
@ -625,6 +581,14 @@ struct xe_device {
atomic64_t global_total_pages;
#endif
/** @psmi: GPU debugging via additional validation HW */
struct {
/** @psmi.capture_obj: PSMI buffer for VRAM */
struct xe_bo *capture_obj[XE_MAX_TILES_PER_DEVICE + 1];
/** @psmi.region_mask: Mask of valid memory regions */
u8 region_mask;
} psmi;
/* private: */
#if IS_ENABLED(CONFIG_DRM_XE_DISPLAY)

4
drivers/gpu/drm/xe/xe_eu_stall.c
View File

@ -649,7 +649,7 @@ static int xe_eu_stall_stream_enable(struct xe_eu_stall_data_stream *stream)
return -ETIMEDOUT;
}
if (XE_WA(gt, 22016596838))
if (XE_GT_WA(gt, 22016596838))
xe_gt_mcr_multicast_write(gt, ROW_CHICKEN2,
_MASKED_BIT_ENABLE(DISABLE_DOP_GATING));
@ -805,7 +805,7 @@ static int xe_eu_stall_disable_locked(struct xe_eu_stall_data_stream *stream)
cancel_delayed_work_sync(&stream->buf_poll_work);
if (XE_WA(gt, 22016596838))
if (XE_GT_WA(gt, 22016596838))
xe_gt_mcr_multicast_write(gt, ROW_CHICKEN2,
_MASKED_BIT_DISABLE(DISABLE_DOP_GATING));

111
drivers/gpu/drm/xe/xe_exec_queue.c
View File

@ -12,6 +12,7 @@
#include <drm/drm_file.h>
#include <uapi/drm/xe_drm.h>
#include "xe_dep_scheduler.h"
#include "xe_device.h"
#include "xe_gt.h"
#include "xe_hw_engine_class_sysfs.h"
@ -39,6 +40,12 @@ static int exec_queue_user_extensions(struct xe_device *xe, struct xe_exec_queue
static void __xe_exec_queue_free(struct xe_exec_queue *q)
{
int i;
for (i = 0; i < XE_EXEC_QUEUE_TLB_INVAL_COUNT; ++i)
if (q->tlb_inval[i].dep_scheduler)
xe_dep_scheduler_fini(q->tlb_inval[i].dep_scheduler);
if (xe_exec_queue_uses_pxp(q))
xe_pxp_exec_queue_remove(gt_to_xe(q->gt)->pxp, q);
if (q->vm)
@ -50,6 +57,39 @@ static void __xe_exec_queue_free(struct xe_exec_queue *q)
kfree(q);
}
static int alloc_dep_schedulers(struct xe_device *xe, struct xe_exec_queue *q)
{
struct xe_tile *tile = gt_to_tile(q->gt);
int i;
for (i = 0; i < XE_EXEC_QUEUE_TLB_INVAL_COUNT; ++i) {
struct xe_dep_scheduler *dep_scheduler;
struct xe_gt *gt;
struct workqueue_struct *wq;
if (i == XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT)
gt = tile->primary_gt;
else
gt = tile->media_gt;
if (!gt)
continue;
wq = gt->tlb_inval.job_wq;
#define MAX_TLB_INVAL_JOBS 16 /* Picking a reasonable value */
dep_scheduler = xe_dep_scheduler_create(xe, wq, q->name,
MAX_TLB_INVAL_JOBS);
if (IS_ERR(dep_scheduler))
return PTR_ERR(dep_scheduler);
q->tlb_inval[i].dep_scheduler = dep_scheduler;
}
#undef MAX_TLB_INVAL_JOBS
return 0;
}
static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe,
struct xe_vm *vm,
u32 logical_mask,
@ -94,6 +134,14 @@ static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe,
else
q->sched_props.priority = XE_EXEC_QUEUE_PRIORITY_NORMAL;
if (q->flags & (EXEC_QUEUE_FLAG_MIGRATE | EXEC_QUEUE_FLAG_VM)) {
err = alloc_dep_schedulers(xe, q);
if (err) {
__xe_exec_queue_free(q);
return ERR_PTR(err);
}
}
if (vm)
q->vm = xe_vm_get(vm);
@ -741,6 +789,21 @@ int xe_exec_queue_get_property_ioctl(struct drm_device *dev, void *data,
return ret;
}
/**
* xe_exec_queue_lrc() - Get the LRC from exec queue.
* @q: The exec_queue.
*
* Retrieves the primary LRC for the exec queue. Note that this function
* returns only the first LRC instance, even when multiple parallel LRCs
* are configured.
*
* Return: Pointer to LRC on success, error on failure
*/
struct xe_lrc *xe_exec_queue_lrc(struct xe_exec_queue *q)
{
return q->lrc[0];
}
/**
* xe_exec_queue_is_lr() - Whether an exec_queue is long-running
* @q: The exec_queue
@ -1028,3 +1091,51 @@ int xe_exec_queue_last_fence_test_dep(struct xe_exec_queue *q, struct xe_vm *vm)
return err;
}
/**
* xe_exec_queue_contexts_hwsp_rebase - Re-compute GGTT references
* within all LRCs of a queue.
* @q: the &xe_exec_queue struct instance containing target LRCs
* @scratch: scratch buffer to be used as temporary storage
*
* Returns: zero on success, negative error code on failure
*/
int xe_exec_queue_contexts_hwsp_rebase(struct xe_exec_queue *q, void *scratch)
{
int i;
int err = 0;
for (i = 0; i < q->width; ++i) {
xe_lrc_update_memirq_regs_with_address(q->lrc[i], q->hwe, scratch);
xe_lrc_update_hwctx_regs_with_address(q->lrc[i]);
err = xe_lrc_setup_wa_bb_with_scratch(q->lrc[i], q->hwe, scratch);
if (err)
break;
}
return err;
}
/**
* xe_exec_queue_jobs_ring_restore - Re-emit ring commands of requests pending on given queue.
* @q: the &xe_exec_queue struct instance
*/
void xe_exec_queue_jobs_ring_restore(struct xe_exec_queue *q)
{
struct xe_gpu_scheduler *sched = &q->guc->sched;
struct xe_sched_job *job;
/*
* This routine is used within VF migration recovery. This means
* using the lock here introduces a restriction: we cannot wait
* for any GFX HW response while the lock is taken.
*/
spin_lock(&sched->base.job_list_lock);
list_for_each_entry(job, &sched->base.pending_list, drm.list) {
if (xe_sched_job_is_error(job))
continue;
q->ring_ops->emit_job(job);
}
spin_unlock(&sched->base.job_list_lock);
}

5
drivers/gpu/drm/xe/xe_exec_queue.h
View File

@ -90,4 +90,9 @@ int xe_exec_queue_last_fence_test_dep(struct xe_exec_queue *q,
struct xe_vm *vm);
void xe_exec_queue_update_run_ticks(struct xe_exec_queue *q);
int xe_exec_queue_contexts_hwsp_rebase(struct xe_exec_queue *q, void *scratch);
void xe_exec_queue_jobs_ring_restore(struct xe_exec_queue *q);
struct xe_lrc *xe_exec_queue_lrc(struct xe_exec_queue *q);
#endif

15
drivers/gpu/drm/xe/xe_exec_queue_types.h
View File

@ -87,6 +87,8 @@ struct xe_exec_queue {
#define EXEC_QUEUE_FLAG_HIGH_PRIORITY BIT(4)
/* flag to indicate low latency hint to guc */
#define EXEC_QUEUE_FLAG_LOW_LATENCY BIT(5)
/* for migration (kernel copy, clear, bind) jobs */
#define EXEC_QUEUE_FLAG_MIGRATE BIT(6)
/**
* @flags: flags for this exec queue, should statically setup aside from ban
@ -132,6 +134,19 @@ struct xe_exec_queue {
struct list_head link;
} lr;
#define XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT 0
#define XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT 1
#define XE_EXEC_QUEUE_TLB_INVAL_COUNT (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT + 1)
/** @tlb_inval: TLB invalidations exec queue state */
struct {
/**
* @tlb_inval.dep_scheduler: The TLB invalidation
* dependency scheduler
*/
struct xe_dep_scheduler *dep_scheduler;
} tlb_inval[XE_EXEC_QUEUE_TLB_INVAL_COUNT];
/** @pxp: PXP info tracking */
struct {
/** @pxp.type: PXP session type used by this queue */

10
drivers/gpu/drm/xe/xe_gen_wa_oob.c
View File

@ -123,11 +123,19 @@ static int parse(FILE *input, FILE *csource, FILE *cheader, char *prefix)
return 0;
}
/* Avoid GNU vs POSIX basename() discrepancy, just use our own */
static const char *xbasename(const char *s)
{
const char *p = strrchr(s, '/');
return p ? p + 1 : s;
}
static int fn_to_prefix(const char *fn, char *prefix, size_t size)
{
size_t len;
fn = basename(fn);
fn = xbasename(fn);
len = strlen(fn);
if (len > size - 1)

15
drivers/gpu/drm/xe/xe_ggtt.c
View File

@ -23,13 +23,13 @@
#include "xe_device.h"
#include "xe_gt.h"
#include "xe_gt_printk.h"
#include "xe_gt_tlb_invalidation.h"
#include "xe_map.h"
#include "xe_mmio.h"
#include "xe_pm.h"
#include "xe_res_cursor.h"
#include "xe_sriov.h"
#include "xe_tile_sriov_vf.h"
#include "xe_tlb_inval.h"
#include "xe_wa.h"
#include "xe_wopcm.h"
@ -106,10 +106,10 @@ static unsigned int probe_gsm_size(struct pci_dev *pdev)
static void ggtt_update_access_counter(struct xe_ggtt *ggtt)
{
struct xe_tile *tile = ggtt->tile;
struct xe_gt *affected_gt = XE_WA(tile->primary_gt, 22019338487) ?
struct xe_gt *affected_gt = XE_GT_WA(tile->primary_gt, 22019338487) ?
tile->primary_gt : tile->media_gt;
struct xe_mmio *mmio = &affected_gt->mmio;
u32 max_gtt_writes = XE_WA(ggtt->tile->primary_gt, 22019338487) ? 1100 : 63;
u32 max_gtt_writes = XE_GT_WA(ggtt->tile->primary_gt, 22019338487) ? 1100 : 63;
/*
* Wa_22019338487: GMD_ID is a RO register, a dummy write forces gunit
* to wait for completion of prior GTT writes before letting this through.
@ -284,8 +284,8 @@ int xe_ggtt_init_early(struct xe_ggtt *ggtt)
if (GRAPHICS_VERx100(xe) >= 1270)
ggtt->pt_ops = (ggtt->tile->media_gt &&
XE_WA(ggtt->tile->media_gt, 22019338487)) ||
XE_WA(ggtt->tile->primary_gt, 22019338487) ?
XE_GT_WA(ggtt->tile->media_gt, 22019338487)) ||
XE_GT_WA(ggtt->tile->primary_gt, 22019338487) ?
&xelpg_pt_wa_ops : &xelpg_pt_ops;
else
ggtt->pt_ops = &xelp_pt_ops;
@ -438,9 +438,8 @@ static void ggtt_invalidate_gt_tlb(struct xe_gt *gt)
if (!gt)
return;
err = xe_gt_tlb_invalidation_ggtt(gt);
if (err)
drm_warn(&gt_to_xe(gt)->drm, "xe_gt_tlb_invalidation_ggtt error=%d", err);
err = xe_tlb_inval_ggtt(&gt->tlb_inval);
xe_gt_WARN(gt, err, "Failed to invalidate GGTT (%pe)", ERR_PTR(err));
}
static void xe_ggtt_invalidate(struct xe_ggtt *ggtt)

13
drivers/gpu/drm/xe/xe_gpu_scheduler.c
View File

@ -101,6 +101,19 @@ void xe_sched_submission_stop(struct xe_gpu_scheduler *sched)
cancel_work_sync(&sched->work_process_msg);
}
/**
* xe_sched_submission_stop_async - Stop further runs of submission tasks on a scheduler.
* @sched: the &xe_gpu_scheduler struct instance
*
* This call disables further runs of scheduling work queue. It does not wait
* for any in-progress runs to finish, only makes sure no further runs happen
* afterwards.
*/
void xe_sched_submission_stop_async(struct xe_gpu_scheduler *sched)
{
drm_sched_wqueue_stop(&sched->base);
}
void xe_sched_submission_resume_tdr(struct xe_gpu_scheduler *sched)
{
drm_sched_resume_timeout(&sched->base, sched->base.timeout);

1
drivers/gpu/drm/xe/xe_gpu_scheduler.h
View File

@ -21,6 +21,7 @@ void xe_sched_fini(struct xe_gpu_scheduler *sched);
void xe_sched_submission_start(struct xe_gpu_scheduler *sched);
void xe_sched_submission_stop(struct xe_gpu_scheduler *sched);
void xe_sched_submission_stop_async(struct xe_gpu_scheduler *sched);
void xe_sched_submission_resume_tdr(struct xe_gpu_scheduler *sched);

6
drivers/gpu/drm/xe/xe_gsc.c
View File

@ -266,7 +266,7 @@ static int gsc_upload_and_init(struct xe_gsc *gsc)
unsigned int fw_ref;
int ret;
if (XE_WA(tile->primary_gt, 14018094691)) {
if (XE_GT_WA(tile->primary_gt, 14018094691)) {
fw_ref = xe_force_wake_get(gt_to_fw(tile->primary_gt), XE_FORCEWAKE_ALL);
/*
@ -281,7 +281,7 @@ static int gsc_upload_and_init(struct xe_gsc *gsc)
ret = gsc_upload(gsc);
if (XE_WA(tile->primary_gt, 14018094691))
if (XE_GT_WA(tile->primary_gt, 14018094691))
xe_force_wake_put(gt_to_fw(tile->primary_gt), fw_ref);
if (ret)
@ -593,7 +593,7 @@ void xe_gsc_wa_14015076503(struct xe_gt *gt, bool prep)
u32 gs1_clr = prep ? 0 : HECI_H_GS1_ER_PREP;
/* WA only applies if the GSC is loaded */
if (!XE_WA(gt, 14015076503) || !gsc_fw_is_loaded(gt))
if (!XE_GT_WA(gt, 14015076503) || !gsc_fw_is_loaded(gt))
return;
xe_mmio_rmw32(&gt->mmio, HECI_H_GS1(MTL_GSC_HECI2_BASE), gs1_clr, gs1_set);

34
drivers/gpu/drm/xe/xe_gt.c
View File

@ -37,10 +37,10 @@
#include "xe_gt_sriov_pf.h"
#include "xe_gt_sriov_vf.h"
#include "xe_gt_sysfs.h"
#include "xe_gt_tlb_invalidation.h"
#include "xe_gt_topology.h"
#include "xe_guc_exec_queue_types.h"
#include "xe_guc_pc.h"
#include "xe_guc_submit.h"
#include "xe_hw_fence.h"
#include "xe_hw_engine_class_sysfs.h"
#include "xe_irq.h"
@ -57,6 +57,7 @@
#include "xe_sa.h"
#include "xe_sched_job.h"
#include "xe_sriov.h"
#include "xe_tlb_inval.h"
#include "xe_tuning.h"
#include "xe_uc.h"
#include "xe_uc_fw.h"
@ -105,7 +106,7 @@ static void xe_gt_enable_host_l2_vram(struct xe_gt *gt)
unsigned int fw_ref;
u32 reg;
if (!XE_WA(gt, 16023588340))
if (!XE_GT_WA(gt, 16023588340))
return;
fw_ref = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
@ -127,7 +128,7 @@ static void xe_gt_disable_host_l2_vram(struct xe_gt *gt)
unsigned int fw_ref;
u32 reg;
if (!XE_WA(gt, 16023588340))
if (!XE_GT_WA(gt, 16023588340))
return;
if (xe_gt_is_media_type(gt))
@ -399,7 +400,7 @@ int xe_gt_init_early(struct xe_gt *gt)
xe_reg_sr_init(&gt->reg_sr, "GT", gt_to_xe(gt));
err = xe_wa_init(gt);
err = xe_wa_gt_init(gt);
if (err)
return err;
@ -407,12 +408,12 @@ int xe_gt_init_early(struct xe_gt *gt)
if (err)
return err;
xe_wa_process_oob(gt);
xe_wa_process_gt_oob(gt);
xe_force_wake_init_gt(gt, gt_to_fw(gt));
spin_lock_init(&gt->global_invl_lock);
err = xe_gt_tlb_invalidation_init_early(gt);
err = xe_gt_tlb_inval_init_early(gt);
if (err)
return err;
@ -564,11 +565,9 @@ static int gt_init_with_all_forcewake(struct xe_gt *gt)
if (xe_gt_is_main_type(gt)) {
struct xe_tile *tile = gt_to_tile(gt);
tile->migrate = xe_migrate_init(tile);
if (IS_ERR(tile->migrate)) {
err = PTR_ERR(tile->migrate);
err = xe_migrate_init(tile->migrate);
if (err)
goto err_force_wake;
}
}
err = xe_uc_load_hw(&gt->uc);
@ -804,6 +803,11 @@ static int do_gt_restart(struct xe_gt *gt)
return 0;
}
static int gt_wait_reset_unblock(struct xe_gt *gt)
{
return xe_guc_wait_reset_unblock(&gt->uc.guc);
}
static int gt_reset(struct xe_gt *gt)
{
unsigned int fw_ref;
@ -818,6 +822,10 @@ static int gt_reset(struct xe_gt *gt)
xe_gt_info(gt, "reset started\n");
err = gt_wait_reset_unblock(gt);
if (!err)
xe_gt_warn(gt, "reset block failed to get lifted");
xe_pm_runtime_get(gt_to_xe(gt));
if (xe_fault_inject_gt_reset()) {
@ -842,7 +850,7 @@ static int gt_reset(struct xe_gt *gt)
xe_uc_stop(&gt->uc);
xe_gt_tlb_invalidation_reset(gt);
xe_tlb_inval_reset(&gt->tlb_inval);
err = do_gt_reset(gt);
if (err)
@ -958,7 +966,7 @@ int xe_gt_sanitize_freq(struct xe_gt *gt)
if ((!xe_uc_fw_is_available(&gt->uc.gsc.fw) ||
xe_uc_fw_is_loaded(&gt->uc.gsc.fw) ||
xe_uc_fw_is_in_error_state(&gt->uc.gsc.fw)) &&
XE_WA(gt, 22019338487))
XE_GT_WA(gt, 22019338487))
ret = xe_guc_pc_restore_stashed_freq(&gt->uc.guc.pc);
return ret;
@ -1056,5 +1064,5 @@ void xe_gt_declare_wedged(struct xe_gt *gt)
xe_gt_assert(gt, gt_to_xe(gt)->wedged.mode);
xe_uc_declare_wedged(&gt->uc);
xe_gt_tlb_invalidation_reset(gt);
xe_tlb_inval_reset(&gt->tlb_inval);
}

56
drivers/gpu/drm/xe/xe_gt_debugfs.c
View File

@ -29,6 +29,7 @@
#include "xe_pm.h"
#include "xe_reg_sr.h"
#include "xe_reg_whitelist.h"
#include "xe_sa.h"
#include "xe_sriov.h"
#include "xe_tuning.h"
#include "xe_uc_debugfs.h"
@ -128,7 +129,34 @@ static int sa_info(struct xe_gt *gt, struct drm_printer *p)
xe_pm_runtime_get(gt_to_xe(gt));
drm_suballoc_dump_debug_info(&tile->mem.kernel_bb_pool->base, p,
tile->mem.kernel_bb_pool->gpu_addr);
xe_sa_manager_gpu_addr(tile->mem.kernel_bb_pool));
xe_pm_runtime_put(gt_to_xe(gt));
return 0;
}
static int sa_info_vf_ccs(struct xe_gt *gt, struct drm_printer *p)
{
struct xe_tile *tile = gt_to_tile(gt);
struct xe_sa_manager *bb_pool;
enum xe_sriov_vf_ccs_rw_ctxs ctx_id;
if (!IS_VF_CCS_READY(gt_to_xe(gt)))
return 0;
xe_pm_runtime_get(gt_to_xe(gt));
for_each_ccs_rw_ctx(ctx_id) {
bb_pool = tile->sriov.vf.ccs[ctx_id].mem.ccs_bb_pool;
if (!bb_pool)
break;
drm_printf(p, "ccs %s bb suballoc info\n", ctx_id ? "write" : "read");
drm_printf(p, "-------------------------\n");
drm_suballoc_dump_debug_info(&bb_pool->base, p, xe_sa_manager_gpu_addr(bb_pool));
drm_puts(p, "\n");
}
xe_pm_runtime_put(gt_to_xe(gt));
return 0;
@ -303,6 +331,13 @@ static const struct drm_info_list vf_safe_debugfs_list[] = {
{"hwconfig", .show = xe_gt_debugfs_simple_show, .data = hwconfig},
};
/*
* only for GT debugfs files which are valid on VF. Not valid on PF.
*/
static const struct drm_info_list vf_only_debugfs_list[] = {
{"sa_info_vf_ccs", .show = xe_gt_debugfs_simple_show, .data = sa_info_vf_ccs},
};
/* everything else should be added here */
static const struct drm_info_list pf_only_debugfs_list[] = {
{"hw_engines", .show = xe_gt_debugfs_simple_show, .data = hw_engines},
@ -388,13 +423,18 @@ void xe_gt_debugfs_register(struct xe_gt *gt)
{
struct xe_device *xe = gt_to_xe(gt);
struct drm_minor *minor = gt_to_xe(gt)->drm.primary;
struct dentry *parent = gt->tile->debugfs;
struct dentry *root;
char symlink[16];
char name[8];
xe_gt_assert(gt, minor->debugfs_root);
if (IS_ERR(parent))
return;
snprintf(name, sizeof(name), "gt%d", gt->info.id);
root = debugfs_create_dir(name, minor->debugfs_root);
root = debugfs_create_dir(name, parent);
if (IS_ERR(root)) {
drm_warn(&xe->drm, "Create GT directory failed");
return;
@ -419,6 +459,11 @@ void xe_gt_debugfs_register(struct xe_gt *gt)
drm_debugfs_create_files(pf_only_debugfs_list,
ARRAY_SIZE(pf_only_debugfs_list),
root, minor);
else
drm_debugfs_create_files(vf_only_debugfs_list,
ARRAY_SIZE(vf_only_debugfs_list),
root, minor);
xe_uc_debugfs_register(&gt->uc, root);
@ -426,4 +471,11 @@ void xe_gt_debugfs_register(struct xe_gt *gt)
xe_gt_sriov_pf_debugfs_register(gt, root);
else if (IS_SRIOV_VF(xe))
xe_gt_sriov_vf_debugfs_register(gt, root);
/*
* Backwards compatibility only: create a link for the legacy clients
* who may expect gt/ directory at the root level, not the tile level.
*/
snprintf(symlink, sizeof(symlink), "tile%u/%s", gt->tile->id, name);
debugfs_create_symlink(name, minor->debugfs_root, symlink);
}

21
drivers/gpu/drm/xe/xe_gt_idle.c
View File

@ -322,15 +322,11 @@ static void gt_idle_fini(void *arg)
{
struct kobject *kobj = arg;
struct xe_gt *gt = kobj_to_gt(kobj->parent);
unsigned int fw_ref;
xe_gt_idle_disable_pg(gt);
if (gt_to_xe(gt)->info.skip_guc_pc) {
fw_ref = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
if (gt_to_xe(gt)->info.skip_guc_pc)
xe_gt_idle_disable_c6(gt);
xe_force_wake_put(gt_to_fw(gt), fw_ref);
}
sysfs_remove_files(kobj, gt_idle_attrs);
kobject_put(kobj);
@ -390,14 +386,23 @@ void xe_gt_idle_enable_c6(struct xe_gt *gt)
RC_CTL_HW_ENABLE | RC_CTL_TO_MODE | RC_CTL_RC6_ENABLE);
}
void xe_gt_idle_disable_c6(struct xe_gt *gt)
int xe_gt_idle_disable_c6(struct xe_gt *gt)
{
unsigned int fw_ref;
xe_device_assert_mem_access(gt_to_xe(gt));
xe_force_wake_assert_held(gt_to_fw(gt), XE_FW_GT);
if (IS_SRIOV_VF(gt_to_xe(gt)))
return;
return 0;
fw_ref = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
if (!fw_ref)
return -ETIMEDOUT;
xe_mmio_write32(&gt->mmio, RC_CONTROL, 0);
xe_mmio_write32(&gt->mmio, RC_STATE, 0);
xe_force_wake_put(gt_to_fw(gt), fw_ref);
return 0;
}

2
drivers/gpu/drm/xe/xe_gt_idle.h
View File

@ -13,7 +13,7 @@ struct xe_gt;
int xe_gt_idle_init(struct xe_gt_idle *gtidle);
void xe_gt_idle_enable_c6(struct xe_gt *gt);
void xe_gt_idle_disable_c6(struct xe_gt *gt);
int xe_gt_idle_disable_c6(struct xe_gt *gt);
void xe_gt_idle_enable_pg(struct xe_gt *gt);
void xe_gt_idle_disable_pg(struct xe_gt *gt);
int xe_gt_idle_pg_print(struct xe_gt *gt, struct drm_printer *p);

4
drivers/gpu/drm/xe/xe_gt_mcr.c
View File

@ -46,8 +46,6 @@
* MCR registers are not available on Virtual Function (VF).
*/
#define STEER_SEMAPHORE XE_REG(0xFD0)
static inline struct xe_reg to_xe_reg(struct xe_reg_mcr reg_mcr)
{
return reg_mcr.__reg;
@ -533,7 +531,7 @@ void xe_gt_mcr_set_implicit_defaults(struct xe_gt *gt)
u32 steer_val = REG_FIELD_PREP(MCR_SLICE_MASK, 0) |
REG_FIELD_PREP(MCR_SUBSLICE_MASK, 2);
xe_mmio_write32(&gt->mmio, MCFG_MCR_SELECTOR, steer_val);
xe_mmio_write32(&gt->mmio, STEER_SEMAPHORE, steer_val);
xe_mmio_write32(&gt->mmio, SF_MCR_SELECTOR, steer_val);
/*
* For GAM registers, all reads should be directed to instance 1

35
drivers/gpu/drm/xe/xe_gt_pagefault.c
View File

@ -16,13 +16,13 @@
#include "xe_gt.h"
#include "xe_gt_printk.h"
#include "xe_gt_stats.h"
#include "xe_gt_tlb_invalidation.h"
#include "xe_guc.h"
#include "xe_guc_ct.h"
#include "xe_migrate.h"
#include "xe_svm.h"
#include "xe_trace_bo.h"
#include "xe_vm.h"
#include "xe_vram_types.h"
struct pagefault {
u64 page_addr;
@ -74,7 +74,7 @@ static bool vma_is_valid(struct xe_tile *tile, struct xe_vma *vma)
}
static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,
bool atomic, unsigned int id)
bool need_vram_move, struct xe_vram_region *vram)
{
struct xe_bo *bo = xe_vma_bo(vma);
struct xe_vm *vm = xe_vma_vm(vma);
@ -84,24 +84,13 @@ static int xe_pf_begin(struct drm_exec *exec, struct xe_vma *vma,
if (err)
return err;
if (atomic && IS_DGFX(vm->xe)) {
if (xe_vma_is_userptr(vma)) {
err = -EACCES;
return err;
}
if (!bo)
return 0;
/* Migrate to VRAM, move should invalidate the VMA first */
err = xe_bo_migrate(bo, XE_PL_VRAM0 + id);
if (err)
return err;
} else if (bo) {
/* Create backing store if needed */
err = xe_bo_validate(bo, vm, true);
if (err)
return err;
}
err = need_vram_move ? xe_bo_migrate(bo, vram->placement) :
xe_bo_validate(bo, vm, true);
return 0;
return err;
}
static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,
@ -112,10 +101,14 @@ static int handle_vma_pagefault(struct xe_gt *gt, struct xe_vma *vma,
struct drm_exec exec;
struct dma_fence *fence;
ktime_t end = 0;
int err;
int err, needs_vram;
lockdep_assert_held_write(&vm->lock);
needs_vram = xe_vma_need_vram_for_atomic(vm->xe, vma, atomic);
if (needs_vram < 0 || (needs_vram && xe_vma_is_userptr(vma)))
return needs_vram < 0 ? needs_vram : -EACCES;
xe_gt_stats_incr(gt, XE_GT_STATS_ID_VMA_PAGEFAULT_COUNT, 1);
xe_gt_stats_incr(gt, XE_GT_STATS_ID_VMA_PAGEFAULT_KB, xe_vma_size(vma) / 1024);
@ -138,7 +131,7 @@ retry_userptr:
/* Lock VM and BOs dma-resv */
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec) {
err = xe_pf_begin(&exec, vma, atomic, tile->id);
err = xe_pf_begin(&exec, vma, needs_vram == 1, tile->mem.vram);
drm_exec_retry_on_contention(&exec);
if (xe_vm_validate_should_retry(&exec, err, &end))
err = -EAGAIN;
@ -573,7 +566,7 @@ static int handle_acc(struct xe_gt *gt, struct acc *acc)
/* Lock VM and BOs dma-resv */
drm_exec_init(&exec, 0, 0);
drm_exec_until_all_locked(&exec) {
ret = xe_pf_begin(&exec, vma, true, tile->id);
ret = xe_pf_begin(&exec, vma, IS_DGFX(vm->xe), tile->mem.vram);
drm_exec_retry_on_contention(&exec);
if (ret)
break;

24
drivers/gpu/drm/xe/xe_gt_sriov_pf.c
View File

@ -55,7 +55,12 @@ static void pf_init_workers(struct xe_gt *gt)
static void pf_fini_workers(struct xe_gt *gt)
{
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
disable_work_sync(&gt->sriov.pf.workers.restart);
if (disable_work_sync(&gt->sriov.pf.workers.restart)) {
xe_gt_sriov_dbg_verbose(gt, "pending restart disabled!\n");
/* release an rpm reference taken on the worker's behalf */
xe_pm_runtime_put(gt_to_xe(gt));
}
}
/**
@ -207,8 +212,11 @@ static void pf_cancel_restart(struct xe_gt *gt)
{
xe_gt_assert(gt, IS_SRIOV_PF(gt_to_xe(gt)));
if (cancel_work_sync(&gt->sriov.pf.workers.restart))
if (cancel_work_sync(&gt->sriov.pf.workers.restart)) {
xe_gt_sriov_dbg_verbose(gt, "pending restart canceled!\n");
/* release an rpm reference taken on the worker's behalf */
xe_pm_runtime_put(gt_to_xe(gt));
}
}
/**
@ -226,9 +234,12 @@ static void pf_restart(struct xe_gt *gt)
{
struct xe_device *xe = gt_to_xe(gt);
xe_pm_runtime_get(xe);
xe_gt_assert(gt, !xe_pm_runtime_suspended(xe));
xe_gt_sriov_pf_config_restart(gt);
xe_gt_sriov_pf_control_restart(gt);
/* release an rpm reference taken on our behalf */
xe_pm_runtime_put(xe);
xe_gt_sriov_dbg(gt, "restart completed\n");
@ -247,8 +258,13 @@ static void pf_queue_restart(struct xe_gt *gt)
xe_gt_assert(gt, IS_SRIOV_PF(xe));
if (!queue_work(xe->sriov.wq, &gt->sriov.pf.workers.restart))
/* take an rpm reference on behalf of the worker */
xe_pm_runtime_get_noresume(xe);
if (!queue_work(xe->sriov.wq, &gt->sriov.pf.workers.restart)) {
xe_gt_sriov_dbg(gt, "restart already in queue!\n");
xe_pm_runtime_put(xe);
}
}
/**

13
drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c
View File

@ -33,6 +33,7 @@
#include "xe_migrate.h"
#include "xe_sriov.h"
#include "xe_ttm_vram_mgr.h"
#include "xe_vram_types.h"
#include "xe_wopcm.h"
#define make_u64_from_u32(hi, lo) ((u64)((u64)(u32)(hi) << 32 | (u32)(lo)))
@ -1433,7 +1434,8 @@ fail:
return err;
}
static void pf_release_vf_config_lmem(struct xe_gt *gt, struct xe_gt_sriov_config *config)
/* Return: %true if there was an LMEM provisioned, %false otherwise */
static bool pf_release_vf_config_lmem(struct xe_gt *gt, struct xe_gt_sriov_config *config)
{
xe_gt_assert(gt, IS_DGFX(gt_to_xe(gt)));
xe_gt_assert(gt, xe_gt_is_main_type(gt));
@ -1442,7 +1444,9 @@ static void pf_release_vf_config_lmem(struct xe_gt *gt, struct xe_gt_sriov_confi
if (config->lmem_obj) {
xe_bo_unpin_map_no_vm(config->lmem_obj);
config->lmem_obj = NULL;
return true;
}
return false;
}
static int pf_provision_vf_lmem(struct xe_gt *gt, unsigned int vfid, u64 size)
@ -1604,7 +1608,7 @@ static u64 pf_query_free_lmem(struct xe_gt *gt)
{
struct xe_tile *tile = gt->tile;
return xe_ttm_vram_get_avail(&tile->mem.vram.ttm.manager);
return xe_ttm_vram_get_avail(&tile->mem.vram->ttm.manager);
}
static u64 pf_query_max_lmem(struct xe_gt *gt)
@ -2020,12 +2024,13 @@ static void pf_release_vf_config(struct xe_gt *gt, unsigned int vfid)
{
struct xe_gt_sriov_config *config = pf_pick_vf_config(gt, vfid);
struct xe_device *xe = gt_to_xe(gt);
bool released;
if (xe_gt_is_main_type(gt)) {
pf_release_vf_config_ggtt(gt, config);
if (IS_DGFX(xe)) {
pf_release_vf_config_lmem(gt, config);
if (xe_device_has_lmtt(xe))
released = pf_release_vf_config_lmem(gt, config);
if (released && xe_device_has_lmtt(xe))
pf_update_vf_lmtt(xe, vfid);
}
}

14
drivers/gpu/drm/xe/xe_gt_sriov_vf.c
View File

@ -25,6 +25,7 @@
#include "xe_guc.h"
#include "xe_guc_hxg_helpers.h"
#include "xe_guc_relay.h"
#include "xe_lrc.h"
#include "xe_mmio.h"
#include "xe_sriov.h"
#include "xe_sriov_vf.h"
@ -750,6 +751,19 @@ failed:
return err;
}
/**
* xe_gt_sriov_vf_default_lrcs_hwsp_rebase - Update GGTT references in HWSP of default LRCs.
* @gt: the &xe_gt struct instance
*/
void xe_gt_sriov_vf_default_lrcs_hwsp_rebase(struct xe_gt *gt)
{
struct xe_hw_engine *hwe;
enum xe_hw_engine_id id;
for_each_hw_engine(hwe, gt, id)
xe_default_lrc_update_memirq_regs_with_address(hwe);
}
/**
* xe_gt_sriov_vf_migrated_event_handler - Start a VF migration recovery,
* or just mark that a GuC is ready for it.

1
drivers/gpu/drm/xe/xe_gt_sriov_vf.h
View File

@ -21,6 +21,7 @@ void xe_gt_sriov_vf_guc_versions(struct xe_gt *gt,
int xe_gt_sriov_vf_query_config(struct xe_gt *gt);
int xe_gt_sriov_vf_connect(struct xe_gt *gt);
int xe_gt_sriov_vf_query_runtime(struct xe_gt *gt);
void xe_gt_sriov_vf_default_lrcs_hwsp_rebase(struct xe_gt *gt);
int xe_gt_sriov_vf_notify_resfix_done(struct xe_gt *gt);
void xe_gt_sriov_vf_migrated_event_handler(struct xe_gt *gt);

596
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c
View File

@ -1,596 +0,0 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2023 Intel Corporation
*/
#include "xe_gt_tlb_invalidation.h"
#include "abi/guc_actions_abi.h"
#include "xe_device.h"
#include "xe_force_wake.h"
#include "xe_gt.h"
#include "xe_gt_printk.h"
#include "xe_guc.h"
#include "xe_guc_ct.h"
#include "xe_gt_stats.h"
#include "xe_mmio.h"
#include "xe_pm.h"
#include "xe_sriov.h"
#include "xe_trace.h"
#include "regs/xe_guc_regs.h"
#define FENCE_STACK_BIT DMA_FENCE_FLAG_USER_BITS
/*
* TLB inval depends on pending commands in the CT queue and then the real
* invalidation time. Double up the time to process full CT queue
* just to be on the safe side.
*/
static long tlb_timeout_jiffies(struct xe_gt *gt)
{
/* this reflects what HW/GuC needs to process TLB inv request */
const long hw_tlb_timeout = HZ / 4;
/* this estimates actual delay caused by the CTB transport */
long delay = xe_guc_ct_queue_proc_time_jiffies(&gt->uc.guc.ct);
return hw_tlb_timeout + 2 * delay;
}
static void xe_gt_tlb_invalidation_fence_fini(struct xe_gt_tlb_invalidation_fence *fence)
{
if (WARN_ON_ONCE(!fence->gt))
return;
xe_pm_runtime_put(gt_to_xe(fence->gt));
fence->gt = NULL; /* fini() should be called once */
}
static void
__invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fence *fence)
{
bool stack = test_bit(FENCE_STACK_BIT, &fence->base.flags);
trace_xe_gt_tlb_invalidation_fence_signal(xe, fence);
xe_gt_tlb_invalidation_fence_fini(fence);
dma_fence_signal(&fence->base);
if (!stack)
dma_fence_put(&fence->base);
}
static void
invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fence *fence)
{
list_del(&fence->link);
__invalidation_fence_signal(xe, fence);
}
void xe_gt_tlb_invalidation_fence_signal(struct xe_gt_tlb_invalidation_fence *fence)
{
if (WARN_ON_ONCE(!fence->gt))
return;
__invalidation_fence_signal(gt_to_xe(fence->gt), fence);
}
static void xe_gt_tlb_fence_timeout(struct work_struct *work)
{
struct xe_gt *gt = container_of(work, struct xe_gt,
tlb_invalidation.fence_tdr.work);
struct xe_device *xe = gt_to_xe(gt);
struct xe_gt_tlb_invalidation_fence *fence, *next;
LNL_FLUSH_WORK(&gt->uc.guc.ct.g2h_worker);
spin_lock_irq(&gt->tlb_invalidation.pending_lock);
list_for_each_entry_safe(fence, next,
&gt->tlb_invalidation.pending_fences, link) {
s64 since_inval_ms = ktime_ms_delta(ktime_get(),
fence->invalidation_time);
if (msecs_to_jiffies(since_inval_ms) < tlb_timeout_jiffies(gt))
break;
trace_xe_gt_tlb_invalidation_fence_timeout(xe, fence);
xe_gt_err(gt, "TLB invalidation fence timeout, seqno=%d recv=%d",
fence->seqno, gt->tlb_invalidation.seqno_recv);
fence->base.error = -ETIME;
invalidation_fence_signal(xe, fence);
}
if (!list_empty(&gt->tlb_invalidation.pending_fences))
queue_delayed_work(system_wq,
&gt->tlb_invalidation.fence_tdr,
tlb_timeout_jiffies(gt));
spin_unlock_irq(&gt->tlb_invalidation.pending_lock);
}
/**
* xe_gt_tlb_invalidation_init_early - Initialize GT TLB invalidation state
* @gt: GT structure
*
* Initialize GT TLB invalidation state, purely software initialization, should
* be called once during driver load.
*
* Return: 0 on success, negative error code on error.
*/
int xe_gt_tlb_invalidation_init_early(struct xe_gt *gt)
{
gt->tlb_invalidation.seqno = 1;
INIT_LIST_HEAD(&gt->tlb_invalidation.pending_fences);
spin_lock_init(&gt->tlb_invalidation.pending_lock);
spin_lock_init(&gt->tlb_invalidation.lock);
INIT_DELAYED_WORK(&gt->tlb_invalidation.fence_tdr,
xe_gt_tlb_fence_timeout);
return 0;
}
/**
* xe_gt_tlb_invalidation_reset - Initialize GT TLB invalidation reset
* @gt: GT structure
*
* Signal any pending invalidation fences, should be called during a GT reset
*/
void xe_gt_tlb_invalidation_reset(struct xe_gt *gt)
{
struct xe_gt_tlb_invalidation_fence *fence, *next;
int pending_seqno;
/*
* we can get here before the CTs are even initialized if we're wedging
* very early, in which case there are not going to be any pending
* fences so we can bail immediately.
*/
if (!xe_guc_ct_initialized(&gt->uc.guc.ct))
return;
/*
* CT channel is already disabled at this point. No new TLB requests can
* appear.
*/
mutex_lock(&gt->uc.guc.ct.lock);
spin_lock_irq(&gt->tlb_invalidation.pending_lock);
cancel_delayed_work(&gt->tlb_invalidation.fence_tdr);
/*
* We might have various kworkers waiting for TLB flushes to complete
* which are not tracked with an explicit TLB fence, however at this
* stage that will never happen since the CT is already disabled, so
* make sure we signal them here under the assumption that we have
* completed a full GT reset.
*/
if (gt->tlb_invalidation.seqno == 1)
pending_seqno = TLB_INVALIDATION_SEQNO_MAX - 1;
else
pending_seqno = gt->tlb_invalidation.seqno - 1;
WRITE_ONCE(gt->tlb_invalidation.seqno_recv, pending_seqno);
list_for_each_entry_safe(fence, next,
&gt->tlb_invalidation.pending_fences, link)
invalidation_fence_signal(gt_to_xe(gt), fence);
spin_unlock_irq(&gt->tlb_invalidation.pending_lock);
mutex_unlock(&gt->uc.guc.ct.lock);
}
static bool tlb_invalidation_seqno_past(struct xe_gt *gt, int seqno)
{
int seqno_recv = READ_ONCE(gt->tlb_invalidation.seqno_recv);
if (seqno - seqno_recv < -(TLB_INVALIDATION_SEQNO_MAX / 2))
return false;
if (seqno - seqno_recv > (TLB_INVALIDATION_SEQNO_MAX / 2))
return true;
return seqno_recv >= seqno;
}
static int send_tlb_invalidation(struct xe_guc *guc,
struct xe_gt_tlb_invalidation_fence *fence,
u32 *action, int len)
{
struct xe_gt *gt = guc_to_gt(guc);
struct xe_device *xe = gt_to_xe(gt);
int seqno;
int ret;
xe_gt_assert(gt, fence);
/*
* XXX: The seqno algorithm relies on TLB invalidation being processed
* in order which they currently are, if that changes the algorithm will
* need to be updated.
*/
mutex_lock(&guc->ct.lock);
seqno = gt->tlb_invalidation.seqno;
fence->seqno = seqno;
trace_xe_gt_tlb_invalidation_fence_send(xe, fence);
action[1] = seqno;
ret = xe_guc_ct_send_locked(&guc->ct, action, len,
G2H_LEN_DW_TLB_INVALIDATE, 1);
if (!ret) {
spin_lock_irq(&gt->tlb_invalidation.pending_lock);
/*
* We haven't actually published the TLB fence as per
* pending_fences, but in theory our seqno could have already
* been written as we acquired the pending_lock. In such a case
* we can just go ahead and signal the fence here.
*/
if (tlb_invalidation_seqno_past(gt, seqno)) {
__invalidation_fence_signal(xe, fence);
} else {
fence->invalidation_time = ktime_get();
list_add_tail(&fence->link,
&gt->tlb_invalidation.pending_fences);
if (list_is_singular(&gt->tlb_invalidation.pending_fences))
queue_delayed_work(system_wq,
&gt->tlb_invalidation.fence_tdr,
tlb_timeout_jiffies(gt));
}
spin_unlock_irq(&gt->tlb_invalidation.pending_lock);
} else {
__invalidation_fence_signal(xe, fence);
}
if (!ret) {
gt->tlb_invalidation.seqno = (gt->tlb_invalidation.seqno + 1) %
TLB_INVALIDATION_SEQNO_MAX;
if (!gt->tlb_invalidation.seqno)
gt->tlb_invalidation.seqno = 1;
}
mutex_unlock(&guc->ct.lock);
xe_gt_stats_incr(gt, XE_GT_STATS_ID_TLB_INVAL, 1);
return ret;
}
#define MAKE_INVAL_OP(type) ((type << XE_GUC_TLB_INVAL_TYPE_SHIFT) | \
XE_GUC_TLB_INVAL_MODE_HEAVY << XE_GUC_TLB_INVAL_MODE_SHIFT | \
XE_GUC_TLB_INVAL_FLUSH_CACHE)
/**
* xe_gt_tlb_invalidation_guc - Issue a TLB invalidation on this GT for the GuC
* @gt: GT structure
* @fence: invalidation fence which will be signal on TLB invalidation
* completion
*
* Issue a TLB invalidation for the GuC. Completion of TLB is asynchronous and
* caller can use the invalidation fence to wait for completion.
*
* Return: 0 on success, negative error code on error
*/
static int xe_gt_tlb_invalidation_guc(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence)
{
u32 action[] = {
XE_GUC_ACTION_TLB_INVALIDATION,
0, /* seqno, replaced in send_tlb_invalidation */
MAKE_INVAL_OP(XE_GUC_TLB_INVAL_GUC),
};
int ret;
ret = send_tlb_invalidation(&gt->uc.guc, fence, action,
ARRAY_SIZE(action));
/*
* -ECANCELED indicates the CT is stopped for a GT reset. TLB caches
* should be nuked on a GT reset so this error can be ignored.
*/
if (ret == -ECANCELED)
return 0;
return ret;
}
/**
* xe_gt_tlb_invalidation_ggtt - Issue a TLB invalidation on this GT for the GGTT
* @gt: GT structure
*
* Issue a TLB invalidation for the GGTT. Completion of TLB invalidation is
* synchronous.
*
* Return: 0 on success, negative error code on error
*/
int xe_gt_tlb_invalidation_ggtt(struct xe_gt *gt)
{
struct xe_device *xe = gt_to_xe(gt);
unsigned int fw_ref;
if (xe_guc_ct_enabled(&gt->uc.guc.ct) &&
gt->uc.guc.submission_state.enabled) {
struct xe_gt_tlb_invalidation_fence fence;
int ret;
xe_gt_tlb_invalidation_fence_init(gt, &fence, true);
ret = xe_gt_tlb_invalidation_guc(gt, &fence);
if (ret)
return ret;
xe_gt_tlb_invalidation_fence_wait(&fence);
} else if (xe_device_uc_enabled(xe) && !xe_device_wedged(xe)) {
struct xe_mmio *mmio = &gt->mmio;
if (IS_SRIOV_VF(xe))
return 0;
fw_ref = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
if (xe->info.platform == XE_PVC || GRAPHICS_VER(xe) >= 20) {
xe_mmio_write32(mmio, PVC_GUC_TLB_INV_DESC1,
PVC_GUC_TLB_INV_DESC1_INVALIDATE);
xe_mmio_write32(mmio, PVC_GUC_TLB_INV_DESC0,
PVC_GUC_TLB_INV_DESC0_VALID);
} else {
xe_mmio_write32(mmio, GUC_TLB_INV_CR,
GUC_TLB_INV_CR_INVALIDATE);
}
xe_force_wake_put(gt_to_fw(gt), fw_ref);
}
return 0;
}
static int send_tlb_invalidation_all(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence)
{
u32 action[] = {
XE_GUC_ACTION_TLB_INVALIDATION_ALL,
0, /* seqno, replaced in send_tlb_invalidation */
MAKE_INVAL_OP(XE_GUC_TLB_INVAL_FULL),
};
return send_tlb_invalidation(&gt->uc.guc, fence, action, ARRAY_SIZE(action));
}
/**
* xe_gt_tlb_invalidation_all - Invalidate all TLBs across PF and all VFs.
* @gt: the &xe_gt structure
* @fence: the &xe_gt_tlb_invalidation_fence to be signaled on completion
*
* Send a request to invalidate all TLBs across PF and all VFs.
*
* Return: 0 on success, negative error code on error
*/
int xe_gt_tlb_invalidation_all(struct xe_gt *gt, struct xe_gt_tlb_invalidation_fence *fence)
{
int err;
xe_gt_assert(gt, gt == fence->gt);
err = send_tlb_invalidation_all(gt, fence);
if (err)
xe_gt_err(gt, "TLB invalidation request failed (%pe)", ERR_PTR(err));
return err;
}
/*
* Ensure that roundup_pow_of_two(length) doesn't overflow.
* Note that roundup_pow_of_two() operates on unsigned long,
* not on u64.
*/
#define MAX_RANGE_TLB_INVALIDATION_LENGTH (rounddown_pow_of_two(ULONG_MAX))
/**
* xe_gt_tlb_invalidation_range - Issue a TLB invalidation on this GT for an
* address range
*
* @gt: GT structure
* @fence: invalidation fence which will be signal on TLB invalidation
* completion
* @start: start address
* @end: end address
* @asid: address space id
*
* Issue a range based TLB invalidation if supported, if not fallback to a full
* TLB invalidation. Completion of TLB is asynchronous and caller can use
* the invalidation fence to wait for completion.
*
* Return: Negative error code on error, 0 on success
*/
int xe_gt_tlb_invalidation_range(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence,
u64 start, u64 end, u32 asid)
{
struct xe_device *xe = gt_to_xe(gt);
#define MAX_TLB_INVALIDATION_LEN 7
u32 action[MAX_TLB_INVALIDATION_LEN];
u64 length = end - start;
int len = 0;
xe_gt_assert(gt, fence);
/* Execlists not supported */
if (gt_to_xe(gt)->info.force_execlist) {
__invalidation_fence_signal(xe, fence);
return 0;
}
action[len++] = XE_GUC_ACTION_TLB_INVALIDATION;
action[len++] = 0; /* seqno, replaced in send_tlb_invalidation */
if (!xe->info.has_range_tlb_invalidation ||
length > MAX_RANGE_TLB_INVALIDATION_LENGTH) {
action[len++] = MAKE_INVAL_OP(XE_GUC_TLB_INVAL_FULL);
} else {
u64 orig_start = start;
u64 align;
if (length < SZ_4K)
length = SZ_4K;
/*
* We need to invalidate a higher granularity if start address
* is not aligned to length. When start is not aligned with
* length we need to find the length large enough to create an
* address mask covering the required range.
*/
align = roundup_pow_of_two(length);
start = ALIGN_DOWN(start, align);
end = ALIGN(end, align);
length = align;
while (start + length < end) {
length <<= 1;
start = ALIGN_DOWN(orig_start, length);
}
/*
* Minimum invalidation size for a 2MB page that the hardware
* expects is 16MB
*/
if (length >= SZ_2M) {
length = max_t(u64, SZ_16M, length);
start = ALIGN_DOWN(orig_start, length);
}
xe_gt_assert(gt, length >= SZ_4K);
xe_gt_assert(gt, is_power_of_2(length));
xe_gt_assert(gt, !(length & GENMASK(ilog2(SZ_16M) - 1,
ilog2(SZ_2M) + 1)));
xe_gt_assert(gt, IS_ALIGNED(start, length));
action[len++] = MAKE_INVAL_OP(XE_GUC_TLB_INVAL_PAGE_SELECTIVE);
action[len++] = asid;
action[len++] = lower_32_bits(start);
action[len++] = upper_32_bits(start);
action[len++] = ilog2(length) - ilog2(SZ_4K);
}
xe_gt_assert(gt, len <= MAX_TLB_INVALIDATION_LEN);
return send_tlb_invalidation(&gt->uc.guc, fence, action, len);
}
/**
* xe_gt_tlb_invalidation_vm - Issue a TLB invalidation on this GT for a VM
* @gt: graphics tile
* @vm: VM to invalidate
*
* Invalidate entire VM's address space
*/
void xe_gt_tlb_invalidation_vm(struct xe_gt *gt, struct xe_vm *vm)
{
struct xe_gt_tlb_invalidation_fence fence;
u64 range = 1ull << vm->xe->info.va_bits;
int ret;
xe_gt_tlb_invalidation_fence_init(gt, &fence, true);
ret = xe_gt_tlb_invalidation_range(gt, &fence, 0, range, vm->usm.asid);
if (ret < 0)
return;
xe_gt_tlb_invalidation_fence_wait(&fence);
}
/**
* xe_guc_tlb_invalidation_done_handler - TLB invalidation done handler
* @guc: guc
* @msg: message indicating TLB invalidation done
* @len: length of message
*
* Parse seqno of TLB invalidation, wake any waiters for seqno, and signal any
* invalidation fences for seqno. Algorithm for this depends on seqno being
* received in-order and asserts this assumption.
*
* Return: 0 on success, -EPROTO for malformed messages.
*/
int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len)
{
struct xe_gt *gt = guc_to_gt(guc);
struct xe_device *xe = gt_to_xe(gt);
struct xe_gt_tlb_invalidation_fence *fence, *next;
unsigned long flags;
if (unlikely(len != 1))
return -EPROTO;
/*
* This can also be run both directly from the IRQ handler and also in
* process_g2h_msg(). Only one may process any individual CT message,
* however the order they are processed here could result in skipping a
* seqno. To handle that we just process all the seqnos from the last
* seqno_recv up to and including the one in msg[0]. The delta should be
* very small so there shouldn't be much of pending_fences we actually
* need to iterate over here.
*
* From GuC POV we expect the seqnos to always appear in-order, so if we
* see something later in the timeline we can be sure that anything
* appearing earlier has already signalled, just that we have yet to
* officially process the CT message like if racing against
* process_g2h_msg().
*/
spin_lock_irqsave(&gt->tlb_invalidation.pending_lock, flags);
if (tlb_invalidation_seqno_past(gt, msg[0])) {
spin_unlock_irqrestore(&gt->tlb_invalidation.pending_lock, flags);
return 0;
}
WRITE_ONCE(gt->tlb_invalidation.seqno_recv, msg[0]);
list_for_each_entry_safe(fence, next,
&gt->tlb_invalidation.pending_fences, link) {
trace_xe_gt_tlb_invalidation_fence_recv(xe, fence);
if (!tlb_invalidation_seqno_past(gt, fence->seqno))
break;
invalidation_fence_signal(xe, fence);
}
if (!list_empty(&gt->tlb_invalidation.pending_fences))
mod_delayed_work(system_wq,
&gt->tlb_invalidation.fence_tdr,
tlb_timeout_jiffies(gt));
else
cancel_delayed_work(&gt->tlb_invalidation.fence_tdr);
spin_unlock_irqrestore(&gt->tlb_invalidation.pending_lock, flags);
return 0;
}
static const char *
invalidation_fence_get_driver_name(struct dma_fence *dma_fence)
{
return "xe";
}
static const char *
invalidation_fence_get_timeline_name(struct dma_fence *dma_fence)
{
return "invalidation_fence";
}
static const struct dma_fence_ops invalidation_fence_ops = {
.get_driver_name = invalidation_fence_get_driver_name,
.get_timeline_name = invalidation_fence_get_timeline_name,
};
/**
* xe_gt_tlb_invalidation_fence_init - Initialize TLB invalidation fence
* @gt: GT
* @fence: TLB invalidation fence to initialize
* @stack: fence is stack variable
*
* Initialize TLB invalidation fence for use. xe_gt_tlb_invalidation_fence_fini
* will be automatically called when fence is signalled (all fences must signal),
* even on error.
*/
void xe_gt_tlb_invalidation_fence_init(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence,
bool stack)
{
xe_pm_runtime_get_noresume(gt_to_xe(gt));
spin_lock_irq(&gt->tlb_invalidation.lock);
dma_fence_init(&fence->base, &invalidation_fence_ops,
&gt->tlb_invalidation.lock,
dma_fence_context_alloc(1), 1);
spin_unlock_irq(&gt->tlb_invalidation.lock);
INIT_LIST_HEAD(&fence->link);
if (stack)
set_bit(FENCE_STACK_BIT, &fence->base.flags);
else
dma_fence_get(&fence->base);
fence->gt = gt;
}

40
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h
View File

@ -1,40 +0,0 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2023 Intel Corporation
*/
#ifndef _XE_GT_TLB_INVALIDATION_H_
#define _XE_GT_TLB_INVALIDATION_H_
#include <linux/types.h>
#include "xe_gt_tlb_invalidation_types.h"
struct xe_gt;
struct xe_guc;
struct xe_vm;
struct xe_vma;
int xe_gt_tlb_invalidation_init_early(struct xe_gt *gt);
void xe_gt_tlb_invalidation_reset(struct xe_gt *gt);
int xe_gt_tlb_invalidation_ggtt(struct xe_gt *gt);
void xe_gt_tlb_invalidation_vm(struct xe_gt *gt, struct xe_vm *vm);
int xe_gt_tlb_invalidation_all(struct xe_gt *gt, struct xe_gt_tlb_invalidation_fence *fence);
int xe_gt_tlb_invalidation_range(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence,
u64 start, u64 end, u32 asid);
int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len);
void xe_gt_tlb_invalidation_fence_init(struct xe_gt *gt,
struct xe_gt_tlb_invalidation_fence *fence,
bool stack);
void xe_gt_tlb_invalidation_fence_signal(struct xe_gt_tlb_invalidation_fence *fence);
static inline void
xe_gt_tlb_invalidation_fence_wait(struct xe_gt_tlb_invalidation_fence *fence)
{
dma_fence_wait(&fence->base, false);
}
#endif /* _XE_GT_TLB_INVALIDATION_ */

32
drivers/gpu/drm/xe/xe_gt_tlb_invalidation_types.h
View File

@ -1,32 +0,0 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2023 Intel Corporation
*/
#ifndef _XE_GT_TLB_INVALIDATION_TYPES_H_
#define _XE_GT_TLB_INVALIDATION_TYPES_H_
#include <linux/dma-fence.h>
struct xe_gt;
/**
* struct xe_gt_tlb_invalidation_fence - XE GT TLB invalidation fence
*
* Optionally passed to xe_gt_tlb_invalidation and will be signaled upon TLB
* invalidation completion.
*/
struct xe_gt_tlb_invalidation_fence {
/** @base: dma fence base */
struct dma_fence base;
/** @gt: GT which fence belong to */
struct xe_gt *gt;
/** @link: link into list of pending tlb fences */
struct list_head link;
/** @seqno: seqno of TLB invalidation to signal fence one */
int seqno;
/** @invalidation_time: time of TLB invalidation */
ktime_t invalidation_time;
};
#endif

2
drivers/gpu/drm/xe/xe_gt_topology.c
View File

@ -138,7 +138,7 @@ load_l3_bank_mask(struct xe_gt *gt, xe_l3_bank_mask_t l3_bank_mask)
* but there's no tracking number assigned yet so we use a custom
* OOB workaround descriptor.
*/
if (XE_WA(gt, no_media_l3))
if (XE_GT_WA(gt, no_media_l3))
return;
if (GRAPHICS_VER(xe) >= 30) {

33
drivers/gpu/drm/xe/xe_gt_types.h
View File

@ -17,6 +17,7 @@
#include "xe_oa_types.h"
#include "xe_reg_sr_types.h"
#include "xe_sa_types.h"
#include "xe_tlb_inval_types.h"
#include "xe_uc_types.h"
struct xe_exec_queue_ops;
@ -185,34 +186,8 @@ struct xe_gt {
struct work_struct worker;
} reset;
/** @tlb_invalidation: TLB invalidation state */
struct {
/** @tlb_invalidation.seqno: TLB invalidation seqno, protected by CT lock */
#define TLB_INVALIDATION_SEQNO_MAX 0x100000
int seqno;
/**
* @tlb_invalidation.seqno_recv: last received TLB invalidation seqno,
* protected by CT lock
*/
int seqno_recv;
/**
* @tlb_invalidation.pending_fences: list of pending fences waiting TLB
* invaliations, protected by CT lock
*/
struct list_head pending_fences;
/**
* @tlb_invalidation.pending_lock: protects @tlb_invalidation.pending_fences
* and updating @tlb_invalidation.seqno_recv.
*/
spinlock_t pending_lock;
/**
* @tlb_invalidation.fence_tdr: schedules a delayed call to
* xe_gt_tlb_fence_timeout after the timeut interval is over.
*/
struct delayed_work fence_tdr;
/** @tlb_invalidation.lock: protects TLB invalidation fences */
spinlock_t lock;
} tlb_invalidation;
/** @tlb_inval: TLB invalidation state */
struct xe_tlb_inval tlb_inval;
/**
* @ccs_mode: Number of compute engines enabled.
@ -411,7 +386,7 @@ struct xe_gt {
unsigned long *oob;
/**
* @wa_active.oob_initialized: mark oob as initialized to help
* detecting misuse of XE_WA() - it can only be called on
* detecting misuse of XE_GT_WA() - it can only be called on
* initialization after OOB WAs have being processed
*/
bool oob_initialized;

43
drivers/gpu/drm/xe/xe_guc.c
View File

@ -16,6 +16,7 @@
#include "regs/xe_guc_regs.h"
#include "regs/xe_irq_regs.h"
#include "xe_bo.h"
#include "xe_configfs.h"
#include "xe_device.h"
#include "xe_force_wake.h"
#include "xe_gt.h"
@ -81,11 +82,15 @@ static u32 guc_ctl_debug_flags(struct xe_guc *guc)
static u32 guc_ctl_feature_flags(struct xe_guc *guc)
{
struct xe_device *xe = guc_to_xe(guc);
u32 flags = GUC_CTL_ENABLE_LITE_RESTORE;
if (!guc_to_xe(guc)->info.skip_guc_pc)
if (!xe->info.skip_guc_pc)
flags |= GUC_CTL_ENABLE_SLPC;
if (xe_configfs_get_psmi_enabled(to_pci_dev(xe->drm.dev)))
flags |= GUC_CTL_ENABLE_PSMI_LOGGING;
return flags;
}
@ -157,7 +162,7 @@ static bool needs_wa_dual_queue(struct xe_gt *gt)
* on RCS and CCSes with different address spaces, which on DG2 is
* required as a WA for an HW bug.
*/
if (XE_WA(gt, 22011391025))
if (XE_GT_WA(gt, 22011391025))
return true;
/*
@ -184,10 +189,10 @@ static u32 guc_ctl_wa_flags(struct xe_guc *guc)
struct xe_gt *gt = guc_to_gt(guc);
u32 flags = 0;
if (XE_WA(gt, 22012773006))
if (XE_GT_WA(gt, 22012773006))
flags |= GUC_WA_POLLCS;
if (XE_WA(gt, 14014475959))
if (XE_GT_WA(gt, 14014475959))
flags |= GUC_WA_HOLD_CCS_SWITCHOUT;
if (needs_wa_dual_queue(gt))
@ -201,19 +206,22 @@ static u32 guc_ctl_wa_flags(struct xe_guc *guc)
if (GRAPHICS_VERx100(xe) < 1270)
flags |= GUC_WA_PRE_PARSER;
if (XE_WA(gt, 22012727170) || XE_WA(gt, 22012727685))
if (XE_GT_WA(gt, 22012727170) || XE_GT_WA(gt, 22012727685))
flags |= GUC_WA_CONTEXT_ISOLATION;
if (XE_WA(gt, 18020744125) &&
if (XE_GT_WA(gt, 18020744125) &&
!xe_hw_engine_mask_per_class(gt, XE_ENGINE_CLASS_RENDER))
flags |= GUC_WA_RCS_REGS_IN_CCS_REGS_LIST;
if (XE_WA(gt, 1509372804))
if (XE_GT_WA(gt, 1509372804))
flags |= GUC_WA_RENDER_RST_RC6_EXIT;
if (XE_WA(gt, 14018913170))
if (XE_GT_WA(gt, 14018913170))
flags |= GUC_WA_ENABLE_TSC_CHECK_ON_RC6;
if (XE_GT_WA(gt, 16023683509))
flags |= GUC_WA_SAVE_RESTORE_MCFG_REG_AT_MC6;
return flags;
}
@ -992,11 +1000,14 @@ static int guc_load_done(u32 status)
case XE_GUC_LOAD_STATUS_GUC_PREPROD_BUILD_MISMATCH:
case XE_GUC_LOAD_STATUS_ERROR_DEVID_INVALID_GUCTYPE:
case XE_GUC_LOAD_STATUS_HWCONFIG_ERROR:
case XE_GUC_LOAD_STATUS_BOOTROM_VERSION_MISMATCH:
case XE_GUC_LOAD_STATUS_DPC_ERROR:
case XE_GUC_LOAD_STATUS_EXCEPTION:
case XE_GUC_LOAD_STATUS_INIT_DATA_INVALID:
case XE_GUC_LOAD_STATUS_MPU_DATA_INVALID:
case XE_GUC_LOAD_STATUS_INIT_MMIO_SAVE_RESTORE_INVALID:
case XE_GUC_LOAD_STATUS_KLV_WORKAROUND_INIT_ERROR:
case XE_GUC_LOAD_STATUS_INVALID_FTR_FLAG:
return -1;
}
@ -1136,17 +1147,29 @@ static void guc_wait_ucode(struct xe_guc *guc)
}
switch (ukernel) {
case XE_GUC_LOAD_STATUS_HWCONFIG_START:
xe_gt_err(gt, "still extracting hwconfig table.\n");
break;
case XE_GUC_LOAD_STATUS_EXCEPTION:
xe_gt_err(gt, "firmware exception. EIP: %#x\n",
xe_mmio_read32(mmio, SOFT_SCRATCH(13)));
break;
case XE_GUC_LOAD_STATUS_INIT_DATA_INVALID:
xe_gt_err(gt, "illegal init/ADS data\n");
break;
case XE_GUC_LOAD_STATUS_INIT_MMIO_SAVE_RESTORE_INVALID:
xe_gt_err(gt, "illegal register in save/restore workaround list\n");
break;
case XE_GUC_LOAD_STATUS_HWCONFIG_START:
xe_gt_err(gt, "still extracting hwconfig table.\n");
case XE_GUC_LOAD_STATUS_KLV_WORKAROUND_INIT_ERROR:
xe_gt_err(gt, "illegal workaround KLV data\n");
break;
case XE_GUC_LOAD_STATUS_INVALID_FTR_FLAG:
xe_gt_err(gt, "illegal feature flag specified\n");
break;
}

123
drivers/gpu/drm/xe/xe_guc_ads.c
View File

@ -247,7 +247,7 @@ static size_t calculate_regset_size(struct xe_gt *gt)
count += ADS_REGSET_EXTRA_MAX * XE_NUM_HW_ENGINES;
if (XE_WA(gt, 1607983814))
if (XE_GT_WA(gt, 1607983814))
count += LNCFCMOCS_REG_COUNT;
return count * sizeof(struct guc_mmio_reg);
@ -284,52 +284,26 @@ static size_t calculate_golden_lrc_size(struct xe_guc_ads *ads)
return total_size;
}
static void guc_waklv_enable_one_word(struct xe_guc_ads *ads,
enum xe_guc_klv_ids klv_id,
u32 value,
u32 *offset, u32 *remain)
static void guc_waklv_enable(struct xe_guc_ads *ads,
u32 data[], u32 data_len_dw,
u32 *offset, u32 *remain,
enum xe_guc_klv_ids klv_id)
{
u32 size;
u32 klv_entry[] = {
/* 16:16 key/length */
FIELD_PREP(GUC_KLV_0_KEY, klv_id) |
FIELD_PREP(GUC_KLV_0_LEN, 1),
value,
/* 1 dword data */
};
size = sizeof(klv_entry);
size_t size = sizeof(u32) * (1 + data_len_dw);
if (*remain < size) {
drm_warn(&ads_to_xe(ads)->drm,
"w/a klv buffer too small to add klv id %d\n", klv_id);
} else {
xe_map_memcpy_to(ads_to_xe(ads), ads_to_map(ads), *offset,
klv_entry, size);
*offset += size;
*remain -= size;
}
}
static void guc_waklv_enable_simple(struct xe_guc_ads *ads,
enum xe_guc_klv_ids klv_id, u32 *offset, u32 *remain)
{
u32 klv_entry[] = {
/* 16:16 key/length */
FIELD_PREP(GUC_KLV_0_KEY, klv_id) |
FIELD_PREP(GUC_KLV_0_LEN, 0),
/* 0 dwords data */
};
u32 size;
size = sizeof(klv_entry);
if (xe_gt_WARN(ads_to_gt(ads), *remain < size,
"w/a klv buffer too small to add klv id %d\n", klv_id))
"w/a klv buffer too small to add klv id 0x%04X\n", klv_id);
return;
}
/* 16:16 key/length */
xe_map_wr(ads_to_xe(ads), ads_to_map(ads), *offset, u32,
FIELD_PREP(GUC_KLV_0_KEY, klv_id) | FIELD_PREP(GUC_KLV_0_LEN, data_len_dw));
/* data_len_dw dwords of data */
xe_map_memcpy_to(ads_to_xe(ads), ads_to_map(ads),
*offset + sizeof(u32), data, data_len_dw * sizeof(u32));
xe_map_memcpy_to(ads_to_xe(ads), ads_to_map(ads), *offset,
klv_entry, size);
*offset += size;
*remain -= size;
}
@ -343,44 +317,51 @@ static void guc_waklv_init(struct xe_guc_ads *ads)
offset = guc_ads_waklv_offset(ads);
remain = guc_ads_waklv_size(ads);
if (XE_WA(gt, 14019882105) || XE_WA(gt, 16021333562))
guc_waklv_enable_simple(ads,
GUC_WORKAROUND_KLV_BLOCK_INTERRUPTS_WHEN_MGSR_BLOCKED,
&offset, &remain);
if (XE_WA(gt, 18024947630))
guc_waklv_enable_simple(ads,
GUC_WORKAROUND_KLV_ID_GAM_PFQ_SHADOW_TAIL_POLLING,
&offset, &remain);
if (XE_WA(gt, 16022287689))
guc_waklv_enable_simple(ads,
GUC_WORKAROUND_KLV_ID_DISABLE_MTP_DURING_ASYNC_COMPUTE,
&offset, &remain);
if (XE_GT_WA(gt, 14019882105) || XE_GT_WA(gt, 16021333562))
guc_waklv_enable(ads, NULL, 0, &offset, &remain,
GUC_WORKAROUND_KLV_BLOCK_INTERRUPTS_WHEN_MGSR_BLOCKED);
if (XE_GT_WA(gt, 18024947630))
guc_waklv_enable(ads, NULL, 0, &offset, &remain,
GUC_WORKAROUND_KLV_ID_GAM_PFQ_SHADOW_TAIL_POLLING);
if (XE_GT_WA(gt, 16022287689))
guc_waklv_enable(ads, NULL, 0, &offset, &remain,
GUC_WORKAROUND_KLV_ID_DISABLE_MTP_DURING_ASYNC_COMPUTE);
if (XE_WA(gt, 14022866841))
guc_waklv_enable_simple(ads,
GUC_WA_KLV_WAKE_POWER_DOMAINS_FOR_OUTBOUND_MMIO,
&offset, &remain);
if (XE_GT_WA(gt, 14022866841))
guc_waklv_enable(ads, NULL, 0, &offset, &remain,
GUC_WA_KLV_WAKE_POWER_DOMAINS_FOR_OUTBOUND_MMIO);
/*
* On RC6 exit, GuC will write register 0xB04 with the default value provided. As of now,
* the default value for this register is determined to be 0xC40. This could change in the
* future, so GuC depends on KMD to send it the correct value.
*/
if (XE_WA(gt, 13011645652))
guc_waklv_enable_one_word(ads,
GUC_WA_KLV_NP_RD_WRITE_TO_CLEAR_RCSM_AT_CGP_LATE_RESTORE,
0xC40,
&offset, &remain);
if (XE_GT_WA(gt, 13011645652)) {
u32 data = 0xC40;
if (XE_WA(gt, 14022293748) || XE_WA(gt, 22019794406))
guc_waklv_enable_simple(ads,
GUC_WORKAROUND_KLV_ID_BACK_TO_BACK_RCS_ENGINE_RESET,
&offset, &remain);
guc_waklv_enable(ads, &data, sizeof(data) / sizeof(u32), &offset, &remain,
GUC_WA_KLV_NP_RD_WRITE_TO_CLEAR_RCSM_AT_CGP_LATE_RESTORE);
}
if (GUC_FIRMWARE_VER(&gt->uc.guc) >= MAKE_GUC_VER(70, 44, 0) && XE_WA(gt, 16026508708))
guc_waklv_enable_simple(ads,
GUC_WA_KLV_RESET_BB_STACK_PTR_ON_VF_SWITCH,
&offset, &remain);
if (XE_GT_WA(gt, 14022293748) || XE_GT_WA(gt, 22019794406))
guc_waklv_enable(ads, NULL, 0, &offset, &remain,
GUC_WORKAROUND_KLV_ID_BACK_TO_BACK_RCS_ENGINE_RESET);
if (GUC_FIRMWARE_VER(&gt->uc.guc) >= MAKE_GUC_VER(70, 44, 0) && XE_GT_WA(gt, 16026508708))
guc_waklv_enable(ads, NULL, 0, &offset, &remain,
GUC_WA_KLV_RESET_BB_STACK_PTR_ON_VF_SWITCH);
if (GUC_FIRMWARE_VER(&gt->uc.guc) >= MAKE_GUC_VER(70, 47, 0) && XE_GT_WA(gt, 16026007364)) {
u32 data[] = {
0x0,
0xF,
};
guc_waklv_enable(ads, data, sizeof(data) / sizeof(u32), &offset, &remain,
GUC_WA_KLV_RESTORE_UNSAVED_MEDIA_CONTROL_REG);
}
if (XE_GT_WA(gt, 14020001231))
guc_waklv_enable(ads, NULL, 0, &offset, &remain,
GUC_WORKAROUND_KLV_DISABLE_PSMI_INTERRUPTS_AT_C6_ENTRY_RESTORE_AT_EXIT);
size = guc_ads_waklv_size(ads) - remain;
if (!size)
@ -784,7 +765,7 @@ static unsigned int guc_mmio_regset_write(struct xe_guc_ads *ads,
guc_mmio_regset_write_one(ads, regset_map, e->reg, count++);
}
if (XE_WA(hwe->gt, 1607983814) && hwe->class == XE_ENGINE_CLASS_RENDER) {
if (XE_GT_WA(hwe->gt, 1607983814) && hwe->class == XE_ENGINE_CLASS_RENDER) {
for (i = 0; i < LNCFCMOCS_REG_COUNT; i++) {
guc_mmio_regset_write_one(ads, regset_map,
XELP_LNCFCMOCS(i), count++);

2
drivers/gpu/drm/xe/xe_guc_buf.c
View File

@ -164,7 +164,7 @@ u64 xe_guc_cache_gpu_addr_from_ptr(struct xe_guc_buf_cache *cache, const void *p
if (offset < 0 || offset + size > cache->sam->base.size)
return 0;
return cache->sam->gpu_addr + offset;
return xe_sa_manager_gpu_addr(cache->sam) + offset;
}
#if IS_BUILTIN(CONFIG_DRM_XE_KUNIT_TEST)

8
drivers/gpu/drm/xe/xe_guc_ct.c
View File

@ -26,11 +26,11 @@
#include "xe_gt_sriov_pf_control.h"
#include "xe_gt_sriov_pf_monitor.h"
#include "xe_gt_sriov_printk.h"
#include "xe_gt_tlb_invalidation.h"
#include "xe_guc.h"
#include "xe_guc_log.h"
#include "xe_guc_relay.h"
#include "xe_guc_submit.h"
#include "xe_guc_tlb_inval.h"
#include "xe_map.h"
#include "xe_pm.h"
#include "xe_trace_guc.h"
@ -1416,8 +1416,7 @@ static int process_g2h_msg(struct xe_guc_ct *ct, u32 *msg, u32 len)
ret = xe_guc_pagefault_handler(guc, payload, adj_len);
break;
case XE_GUC_ACTION_TLB_INVALIDATION_DONE:
ret = xe_guc_tlb_invalidation_done_handler(guc, payload,
adj_len);
ret = xe_guc_tlb_inval_done_handler(guc, payload, adj_len);
break;
case XE_GUC_ACTION_ACCESS_COUNTER_NOTIFY:
ret = xe_guc_access_counter_notify_handler(guc, payload,
@ -1618,8 +1617,7 @@ static void g2h_fast_path(struct xe_guc_ct *ct, u32 *msg, u32 len)
break;
case XE_GUC_ACTION_TLB_INVALIDATION_DONE:
__g2h_release_space(ct, len);
ret = xe_guc_tlb_invalidation_done_handler(guc, payload,
adj_len);
ret = xe_guc_tlb_inval_done_handler(guc, payload, adj_len);
break;
default:
xe_gt_warn(gt, "NOT_POSSIBLE");

7
drivers/gpu/drm/xe/xe_guc_fwif.h
View File

@ -45,6 +45,11 @@
#define GUC_MAX_ENGINE_CLASSES 16
#define GUC_MAX_INSTANCES_PER_CLASS 32
#define GUC_CONTEXT_NORMAL 0
#define GUC_CONTEXT_COMPRESSION_SAVE 1
#define GUC_CONTEXT_COMPRESSION_RESTORE 2
#define GUC_CONTEXT_COUNT (GUC_CONTEXT_COMPRESSION_RESTORE + 1)
/* Helper for context registration H2G */
struct guc_ctxt_registration_info {
u32 flags;
@ -103,10 +108,12 @@ struct guc_update_exec_queue_policy {
#define GUC_WA_RENDER_RST_RC6_EXIT BIT(19)
#define GUC_WA_RCS_REGS_IN_CCS_REGS_LIST BIT(21)
#define GUC_WA_ENABLE_TSC_CHECK_ON_RC6 BIT(22)
#define GUC_WA_SAVE_RESTORE_MCFG_REG_AT_MC6 BIT(25)
#define GUC_CTL_FEATURE 2
#define GUC_CTL_ENABLE_SLPC BIT(2)
#define GUC_CTL_ENABLE_LITE_RESTORE BIT(4)
#define GUC_CTL_ENABLE_PSMI_LOGGING BIT(7)
#define GUC_CTL_DISABLE_SCHEDULER BIT(14)
#define GUC_CTL_DEBUG 3

23
drivers/gpu/drm/xe/xe_guc_pc.c
View File

@ -722,7 +722,7 @@ static int xe_guc_pc_set_max_freq_locked(struct xe_guc_pc *pc, u32 freq)
*/
int xe_guc_pc_set_max_freq(struct xe_guc_pc *pc, u32 freq)
{
if (XE_WA(pc_to_gt(pc), 22019338487)) {
if (XE_GT_WA(pc_to_gt(pc), 22019338487)) {
if (wait_for_flush_complete(pc) != 0)
return -EAGAIN;
}
@ -835,7 +835,7 @@ static u32 pc_max_freq_cap(struct xe_guc_pc *pc)
{
struct xe_gt *gt = pc_to_gt(pc);
if (XE_WA(gt, 22019338487)) {
if (XE_GT_WA(gt, 22019338487)) {
if (xe_gt_is_media_type(gt))
return min(LNL_MERT_FREQ_CAP, pc->rp0_freq);
else
@ -899,7 +899,7 @@ static int pc_adjust_freq_bounds(struct xe_guc_pc *pc)
if (pc_get_min_freq(pc) > pc->rp0_freq)
ret = pc_set_min_freq(pc, pc->rp0_freq);
if (XE_WA(tile->primary_gt, 14022085890))
if (XE_GT_WA(tile->primary_gt, 14022085890))
ret = pc_set_min_freq(pc, max(BMG_MIN_FREQ, pc_get_min_freq(pc)));
out:
@ -931,7 +931,7 @@ static bool needs_flush_freq_limit(struct xe_guc_pc *pc)
{
struct xe_gt *gt = pc_to_gt(pc);
return XE_WA(gt, 22019338487) &&
return XE_GT_WA(gt, 22019338487) &&
pc->rp0_freq > BMG_MERT_FLUSH_FREQ_CAP;
}
@ -1017,7 +1017,7 @@ static int pc_set_mert_freq_cap(struct xe_guc_pc *pc)
{
int ret;
if (!XE_WA(pc_to_gt(pc), 22019338487))
if (!XE_GT_WA(pc_to_gt(pc), 22019338487))
return 0;
guard(mutex)(&pc->freq_lock);
@ -1076,7 +1076,6 @@ int xe_guc_pc_gucrc_disable(struct xe_guc_pc *pc)
{
struct xe_device *xe = pc_to_xe(pc);
struct xe_gt *gt = pc_to_gt(pc);
unsigned int fw_ref;
int ret = 0;
if (xe->info.skip_guc_pc)
@ -1086,17 +1085,7 @@ int xe_guc_pc_gucrc_disable(struct xe_guc_pc *pc)
if (ret)
return ret;
fw_ref = xe_force_wake_get(gt_to_fw(gt), XE_FORCEWAKE_ALL);
if (!xe_force_wake_ref_has_domain(fw_ref, XE_FORCEWAKE_ALL)) {
xe_force_wake_put(gt_to_fw(gt), fw_ref);
return -ETIMEDOUT;
}
xe_gt_idle_disable_c6(gt);
xe_force_wake_put(gt_to_fw(gt), fw_ref);
return 0;
return xe_gt_idle_disable_c6(gt);
}
/**

209
drivers/gpu/drm/xe/xe_guc_submit.c
View File

@ -542,7 +542,7 @@ static void __register_exec_queue(struct xe_guc *guc,
xe_guc_ct_send(&guc->ct, action, ARRAY_SIZE(action), 0, 0);
}
static void register_exec_queue(struct xe_exec_queue *q)
static void register_exec_queue(struct xe_exec_queue *q, int ctx_type)
{
struct xe_guc *guc = exec_queue_to_guc(q);
struct xe_device *xe = guc_to_xe(guc);
@ -550,6 +550,7 @@ static void register_exec_queue(struct xe_exec_queue *q)
struct guc_ctxt_registration_info info;
xe_gt_assert(guc_to_gt(guc), !exec_queue_registered(q));
xe_gt_assert(guc_to_gt(guc), ctx_type < GUC_CONTEXT_COUNT);
memset(&info, 0, sizeof(info));
info.context_idx = q->guc->id;
@ -559,6 +560,9 @@ static void register_exec_queue(struct xe_exec_queue *q)
info.hwlrca_hi = upper_32_bits(xe_lrc_descriptor(lrc));
info.flags = CONTEXT_REGISTRATION_FLAG_KMD;
if (ctx_type != GUC_CONTEXT_NORMAL)
info.flags |= BIT(ctx_type);
if (xe_exec_queue_is_parallel(q)) {
u64 ggtt_addr = xe_lrc_parallel_ggtt_addr(lrc);
struct iosys_map map = xe_lrc_parallel_map(lrc);
@ -667,12 +671,18 @@ static void wq_item_append(struct xe_exec_queue *q)
if (wq_wait_for_space(q, wqi_size))
return;
xe_gt_assert(guc_to_gt(guc), i == XE_GUC_CONTEXT_WQ_HEADER_DATA_0_TYPE_LEN);
wqi[i++] = FIELD_PREP(WQ_TYPE_MASK, WQ_TYPE_MULTI_LRC) |
FIELD_PREP(WQ_LEN_MASK, len_dw);
xe_gt_assert(guc_to_gt(guc), i == XE_GUC_CONTEXT_WQ_EL_INFO_DATA_1_CTX_DESC_LOW);
wqi[i++] = xe_lrc_descriptor(q->lrc[0]);
xe_gt_assert(guc_to_gt(guc), i ==
XE_GUC_CONTEXT_WQ_EL_INFO_DATA_2_GUCCTX_RINGTAIL_FREEZEPOCS);
wqi[i++] = FIELD_PREP(WQ_GUC_ID_MASK, q->guc->id) |
FIELD_PREP(WQ_RING_TAIL_MASK, q->lrc[0]->ring.tail / sizeof(u64));
xe_gt_assert(guc_to_gt(guc), i == XE_GUC_CONTEXT_WQ_EL_INFO_DATA_3_WI_FENCE_ID);
wqi[i++] = 0;
xe_gt_assert(guc_to_gt(guc), i == XE_GUC_CONTEXT_WQ_EL_CHILD_LIST_DATA_4_RINGTAIL);
for (j = 1; j < q->width; ++j) {
struct xe_lrc *lrc = q->lrc[j];
@ -693,6 +703,50 @@ static void wq_item_append(struct xe_exec_queue *q)
parallel_write(xe, map, wq_desc.tail, q->guc->wqi_tail);
}
static int wq_items_rebase(struct xe_exec_queue *q)
{
struct xe_guc *guc = exec_queue_to_guc(q);
struct xe_device *xe = guc_to_xe(guc);
struct iosys_map map = xe_lrc_parallel_map(q->lrc[0]);
int i = q->guc->wqi_head;
/* the ring starts after a header struct */
iosys_map_incr(&map, offsetof(struct guc_submit_parallel_scratch, wq[0]));
while ((i % WQ_SIZE) != (q->guc->wqi_tail % WQ_SIZE)) {
u32 len_dw, type, val;
if (drm_WARN_ON_ONCE(&xe->drm, i < 0 || i > 2 * WQ_SIZE))
break;
val = xe_map_rd_ring_u32(xe, &map, i / sizeof(u32) +
XE_GUC_CONTEXT_WQ_HEADER_DATA_0_TYPE_LEN,
WQ_SIZE / sizeof(u32));
len_dw = FIELD_GET(WQ_LEN_MASK, val);
type = FIELD_GET(WQ_TYPE_MASK, val);
if (drm_WARN_ON_ONCE(&xe->drm, len_dw >= WQ_SIZE / sizeof(u32)))
break;
if (type == WQ_TYPE_MULTI_LRC) {
val = xe_lrc_descriptor(q->lrc[0]);
xe_map_wr_ring_u32(xe, &map, i / sizeof(u32) +
XE_GUC_CONTEXT_WQ_EL_INFO_DATA_1_CTX_DESC_LOW,
WQ_SIZE / sizeof(u32), val);
} else if (drm_WARN_ON_ONCE(&xe->drm, type != WQ_TYPE_NOOP)) {
break;
}
i += (len_dw + 1) * sizeof(u32);
}
if ((i % WQ_SIZE) != (q->guc->wqi_tail % WQ_SIZE)) {
xe_gt_err(q->gt, "Exec queue fixups incomplete - wqi parse failed\n");
return -EBADMSG;
}
return 0;
}
#define RESUME_PENDING ~0x0ull
static void submit_exec_queue(struct xe_exec_queue *q)
{
@ -761,7 +815,7 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)
if (!exec_queue_killed_or_banned_or_wedged(q) && !xe_sched_job_is_error(job)) {
if (!exec_queue_registered(q))
register_exec_queue(q);
register_exec_queue(q, GUC_CONTEXT_NORMAL);
if (!lr) /* LR jobs are emitted in the exec IOCTL */
q->ring_ops->emit_job(job);
submit_exec_queue(q);
@ -777,6 +831,30 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)
return fence;
}
/**
* xe_guc_jobs_ring_rebase - Re-emit ring commands of requests pending
* on all queues under a guc.
* @guc: the &xe_guc struct instance
*/
void xe_guc_jobs_ring_rebase(struct xe_guc *guc)
{
struct xe_exec_queue *q;
unsigned long index;
/*
* This routine is used within VF migration recovery. This means
* using the lock here introduces a restriction: we cannot wait
* for any GFX HW response while the lock is taken.
*/
mutex_lock(&guc->submission_state.lock);
xa_for_each(&guc->submission_state.exec_queue_lookup, index, q) {
if (exec_queue_killed_or_banned_or_wedged(q))
continue;
xe_exec_queue_jobs_ring_restore(q);
}
mutex_unlock(&guc->submission_state.lock);
}
static void guc_exec_queue_free_job(struct drm_sched_job *drm_job)
{
struct xe_sched_job *job = to_xe_sched_job(drm_job);
@ -1773,6 +1851,43 @@ static void guc_exec_queue_stop(struct xe_guc *guc, struct xe_exec_queue *q)
}
}
/**
* xe_guc_submit_reset_block - Disallow reset calls on given GuC.
* @guc: the &xe_guc struct instance
*/
int xe_guc_submit_reset_block(struct xe_guc *guc)
{
return atomic_fetch_or(1, &guc->submission_state.reset_blocked);
}
/**
* xe_guc_submit_reset_unblock - Allow back reset calls on given GuC.
* @guc: the &xe_guc struct instance
*/
void xe_guc_submit_reset_unblock(struct xe_guc *guc)
{
atomic_set_release(&guc->submission_state.reset_blocked, 0);
wake_up_all(&guc->ct.wq);
}
static int guc_submit_reset_is_blocked(struct xe_guc *guc)
{
return atomic_read_acquire(&guc->submission_state.reset_blocked);
}
/* Maximum time of blocking reset */
#define RESET_BLOCK_PERIOD_MAX (HZ * 5)
/**
* xe_guc_wait_reset_unblock - Wait until reset blocking flag is lifted, or timeout.
* @guc: the &xe_guc struct instance
*/
int xe_guc_wait_reset_unblock(struct xe_guc *guc)
{
return wait_event_timeout(guc->ct.wq,
!guc_submit_reset_is_blocked(guc), RESET_BLOCK_PERIOD_MAX);
}
int xe_guc_submit_reset_prepare(struct xe_guc *guc)
{
int ret;
@ -1826,6 +1941,19 @@ void xe_guc_submit_stop(struct xe_guc *guc)
}
/**
* xe_guc_submit_pause - Stop further runs of submission tasks on given GuC.
* @guc: the &xe_guc struct instance whose scheduler is to be disabled
*/
void xe_guc_submit_pause(struct xe_guc *guc)
{
struct xe_exec_queue *q;
unsigned long index;
xa_for_each(&guc->submission_state.exec_queue_lookup, index, q)
xe_sched_submission_stop_async(&q->guc->sched);
}
static void guc_exec_queue_start(struct xe_exec_queue *q)
{
struct xe_gpu_scheduler *sched = &q->guc->sched;
@ -1866,6 +1994,28 @@ int xe_guc_submit_start(struct xe_guc *guc)
return 0;
}
static void guc_exec_queue_unpause(struct xe_exec_queue *q)
{
struct xe_gpu_scheduler *sched = &q->guc->sched;
xe_sched_submission_start(sched);
}
/**
* xe_guc_submit_unpause - Allow further runs of submission tasks on given GuC.
* @guc: the &xe_guc struct instance whose scheduler is to be enabled
*/
void xe_guc_submit_unpause(struct xe_guc *guc)
{
struct xe_exec_queue *q;
unsigned long index;
xa_for_each(&guc->submission_state.exec_queue_lookup, index, q)
guc_exec_queue_unpause(q);
wake_up_all(&guc->ct.wq);
}
static struct xe_exec_queue *
g2h_exec_queue_lookup(struct xe_guc *guc, u32 guc_id)
{
@ -2377,6 +2527,32 @@ static void guc_exec_queue_print(struct xe_exec_queue *q, struct drm_printer *p)
xe_guc_exec_queue_snapshot_free(snapshot);
}
/**
* xe_guc_register_exec_queue - Register exec queue for a given context type.
* @q: Execution queue
* @ctx_type: Type of the context
*
* This function registers the execution queue with the guc. Special context
* types like GUC_CONTEXT_COMPRESSION_SAVE and GUC_CONTEXT_COMPRESSION_RESTORE
* are only applicable for IGPU and in the VF.
* Submits the execution queue to GUC after registering it.
*
* Returns - None.
*/
void xe_guc_register_exec_queue(struct xe_exec_queue *q, int ctx_type)
{
struct xe_guc *guc = exec_queue_to_guc(q);
struct xe_device *xe = guc_to_xe(guc);
xe_assert(xe, IS_SRIOV_VF(xe));
xe_assert(xe, !IS_DGFX(xe));
xe_assert(xe, (ctx_type > GUC_CONTEXT_NORMAL &&
ctx_type < GUC_CONTEXT_COUNT));
register_exec_queue(q, ctx_type);
enable_scheduling(q);
}
/**
* xe_guc_submit_print - GuC Submit Print.
* @guc: GuC.
@ -2397,3 +2573,32 @@ void xe_guc_submit_print(struct xe_guc *guc, struct drm_printer *p)
guc_exec_queue_print(q, p);
mutex_unlock(&guc->submission_state.lock);
}
/**
* xe_guc_contexts_hwsp_rebase - Re-compute GGTT references within all
* exec queues registered to given GuC.
* @guc: the &xe_guc struct instance
* @scratch: scratch buffer to be used as temporary storage
*
* Returns: zero on success, negative error code on failure.
*/
int xe_guc_contexts_hwsp_rebase(struct xe_guc *guc, void *scratch)
{
struct xe_exec_queue *q;
unsigned long index;
int err = 0;
mutex_lock(&guc->submission_state.lock);
xa_for_each(&guc->submission_state.exec_queue_lookup, index, q) {
err = xe_exec_queue_contexts_hwsp_rebase(q, scratch);
if (err)
break;
if (xe_exec_queue_is_parallel(q))
err = wq_items_rebase(q);
if (err)
break;
}
mutex_unlock(&guc->submission_state.lock);
return err;
}

10
drivers/gpu/drm/xe/xe_guc_submit.h
View File

@ -18,6 +18,11 @@ int xe_guc_submit_reset_prepare(struct xe_guc *guc);
void xe_guc_submit_reset_wait(struct xe_guc *guc);
void xe_guc_submit_stop(struct xe_guc *guc);
int xe_guc_submit_start(struct xe_guc *guc);
void xe_guc_submit_pause(struct xe_guc *guc);
void xe_guc_submit_unpause(struct xe_guc *guc);
int xe_guc_submit_reset_block(struct xe_guc *guc);
void xe_guc_submit_reset_unblock(struct xe_guc *guc);
int xe_guc_wait_reset_unblock(struct xe_guc *guc);
void xe_guc_submit_wedge(struct xe_guc *guc);
int xe_guc_read_stopped(struct xe_guc *guc);
@ -29,6 +34,8 @@ int xe_guc_exec_queue_memory_cat_error_handler(struct xe_guc *guc, u32 *msg,
int xe_guc_exec_queue_reset_failure_handler(struct xe_guc *guc, u32 *msg, u32 len);
int xe_guc_error_capture_handler(struct xe_guc *guc, u32 *msg, u32 len);
void xe_guc_jobs_ring_rebase(struct xe_guc *guc);
struct xe_guc_submit_exec_queue_snapshot *
xe_guc_exec_queue_snapshot_capture(struct xe_exec_queue *q);
void
@ -39,5 +46,8 @@ xe_guc_exec_queue_snapshot_print(struct xe_guc_submit_exec_queue_snapshot *snaps
void
xe_guc_exec_queue_snapshot_free(struct xe_guc_submit_exec_queue_snapshot *snapshot);
void xe_guc_submit_print(struct xe_guc *guc, struct drm_printer *p);
void xe_guc_register_exec_queue(struct xe_exec_queue *q, int ctx_type);
int xe_guc_contexts_hwsp_rebase(struct xe_guc *guc, void *scratch);
#endif

242
drivers/gpu/drm/xe/xe_guc_tlb_inval.c Normal file
View File

@ -0,0 +1,242 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2025 Intel Corporation
*/
#include "abi/guc_actions_abi.h"
#include "xe_device.h"
#include "xe_gt_stats.h"
#include "xe_gt_types.h"
#include "xe_guc.h"
#include "xe_guc_ct.h"
#include "xe_guc_tlb_inval.h"
#include "xe_force_wake.h"
#include "xe_mmio.h"
#include "xe_tlb_inval.h"
#include "regs/xe_guc_regs.h"
/*
* XXX: The seqno algorithm relies on TLB invalidation being processed in order
* which they currently are by the GuC, if that changes the algorithm will need
* to be updated.
*/
static int send_tlb_inval(struct xe_guc *guc, const u32 *action, int len)
{
struct xe_gt *gt = guc_to_gt(guc);
xe_gt_assert(gt, action[1]); /* Seqno */
xe_gt_stats_incr(gt, XE_GT_STATS_ID_TLB_INVAL, 1);
return xe_guc_ct_send(&guc->ct, action, len,
G2H_LEN_DW_TLB_INVALIDATE, 1);
}
#define MAKE_INVAL_OP(type) ((type << XE_GUC_TLB_INVAL_TYPE_SHIFT) | \
XE_GUC_TLB_INVAL_MODE_HEAVY << XE_GUC_TLB_INVAL_MODE_SHIFT | \
XE_GUC_TLB_INVAL_FLUSH_CACHE)
static int send_tlb_inval_all(struct xe_tlb_inval *tlb_inval, u32 seqno)
{
struct xe_guc *guc = tlb_inval->private;
u32 action[] = {
XE_GUC_ACTION_TLB_INVALIDATION_ALL,
seqno,
MAKE_INVAL_OP(XE_GUC_TLB_INVAL_FULL),
};
return send_tlb_inval(guc, action, ARRAY_SIZE(action));
}
static int send_tlb_inval_ggtt(struct xe_tlb_inval *tlb_inval, u32 seqno)
{
struct xe_guc *guc = tlb_inval->private;
struct xe_gt *gt = guc_to_gt(guc);
struct xe_device *xe = guc_to_xe(guc);
/*
* Returning -ECANCELED in this function is squashed at the caller and
* signals waiters.
*/
if (xe_guc_ct_enabled(&guc->ct) && guc->submission_state.enabled) {
u32 action[] = {
XE_GUC_ACTION_TLB_INVALIDATION,
seqno,
MAKE_INVAL_OP(XE_GUC_TLB_INVAL_GUC),
};
return send_tlb_inval(guc, action, ARRAY_SIZE(action));
} else if (xe_device_uc_enabled(xe) && !xe_device_wedged(xe)) {
struct xe_mmio *mmio = &gt->mmio;
unsigned int fw_ref;
if (IS_SRIOV_VF(xe))
return -ECANCELED;
fw_ref = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
if (xe->info.platform == XE_PVC || GRAPHICS_VER(xe) >= 20) {
xe_mmio_write32(mmio, PVC_GUC_TLB_INV_DESC1,
PVC_GUC_TLB_INV_DESC1_INVALIDATE);
xe_mmio_write32(mmio, PVC_GUC_TLB_INV_DESC0,
PVC_GUC_TLB_INV_DESC0_VALID);
} else {
xe_mmio_write32(mmio, GUC_TLB_INV_CR,
GUC_TLB_INV_CR_INVALIDATE);
}
xe_force_wake_put(gt_to_fw(gt), fw_ref);
}
return -ECANCELED;
}
/*
* Ensure that roundup_pow_of_two(length) doesn't overflow.
* Note that roundup_pow_of_two() operates on unsigned long,
* not on u64.
*/
#define MAX_RANGE_TLB_INVALIDATION_LENGTH (rounddown_pow_of_two(ULONG_MAX))
static int send_tlb_inval_ppgtt(struct xe_tlb_inval *tlb_inval, u32 seqno,
u64 start, u64 end, u32 asid)
{
#define MAX_TLB_INVALIDATION_LEN 7
struct xe_guc *guc = tlb_inval->private;
struct xe_gt *gt = guc_to_gt(guc);
u32 action[MAX_TLB_INVALIDATION_LEN];
u64 length = end - start;
int len = 0;
if (guc_to_xe(guc)->info.force_execlist)
return -ECANCELED;
action[len++] = XE_GUC_ACTION_TLB_INVALIDATION;
action[len++] = seqno;
if (!gt_to_xe(gt)->info.has_range_tlb_inval ||
length > MAX_RANGE_TLB_INVALIDATION_LENGTH) {
action[len++] = MAKE_INVAL_OP(XE_GUC_TLB_INVAL_FULL);
} else {
u64 orig_start = start;
u64 align;
if (length < SZ_4K)
length = SZ_4K;
/*
* We need to invalidate a higher granularity if start address
* is not aligned to length. When start is not aligned with
* length we need to find the length large enough to create an
* address mask covering the required range.
*/
align = roundup_pow_of_two(length);
start = ALIGN_DOWN(start, align);
end = ALIGN(end, align);
length = align;
while (start + length < end) {
length <<= 1;
start = ALIGN_DOWN(orig_start, length);
}
/*
* Minimum invalidation size for a 2MB page that the hardware
* expects is 16MB
*/
if (length >= SZ_2M) {
length = max_t(u64, SZ_16M, length);
start = ALIGN_DOWN(orig_start, length);
}
xe_gt_assert(gt, length >= SZ_4K);
xe_gt_assert(gt, is_power_of_2(length));
xe_gt_assert(gt, !(length & GENMASK(ilog2(SZ_16M) - 1,
ilog2(SZ_2M) + 1)));
xe_gt_assert(gt, IS_ALIGNED(start, length));
action[len++] = MAKE_INVAL_OP(XE_GUC_TLB_INVAL_PAGE_SELECTIVE);
action[len++] = asid;
action[len++] = lower_32_bits(start);
action[len++] = upper_32_bits(start);
action[len++] = ilog2(length) - ilog2(SZ_4K);
}
xe_gt_assert(gt, len <= MAX_TLB_INVALIDATION_LEN);
return send_tlb_inval(guc, action, len);
}
static bool tlb_inval_initialized(struct xe_tlb_inval *tlb_inval)
{
struct xe_guc *guc = tlb_inval->private;
return xe_guc_ct_initialized(&guc->ct);
}
static void tlb_inval_flush(struct xe_tlb_inval *tlb_inval)
{
struct xe_guc *guc = tlb_inval->private;
LNL_FLUSH_WORK(&guc->ct.g2h_worker);
}
static long tlb_inval_timeout_delay(struct xe_tlb_inval *tlb_inval)
{
struct xe_guc *guc = tlb_inval->private;
/* this reflects what HW/GuC needs to process TLB inv request */
const long hw_tlb_timeout = HZ / 4;
/* this estimates actual delay caused by the CTB transport */
long delay = xe_guc_ct_queue_proc_time_jiffies(&guc->ct);
return hw_tlb_timeout + 2 * delay;
}
static const struct xe_tlb_inval_ops guc_tlb_inval_ops = {
.all = send_tlb_inval_all,
.ggtt = send_tlb_inval_ggtt,
.ppgtt = send_tlb_inval_ppgtt,
.initialized = tlb_inval_initialized,
.flush = tlb_inval_flush,
.timeout_delay = tlb_inval_timeout_delay,
};
/**
* xe_guc_tlb_inval_init_early() - Init GuC TLB invalidation early
* @guc: GuC object
* @tlb_inval: TLB invalidation client
*
* Inititialize GuC TLB invalidation by setting back pointer in TLB invalidation
* client to the GuC and setting GuC backend ops.
*/
void xe_guc_tlb_inval_init_early(struct xe_guc *guc,
struct xe_tlb_inval *tlb_inval)
{
tlb_inval->private = guc;
tlb_inval->ops = &guc_tlb_inval_ops;
}
/**
* xe_guc_tlb_inval_done_handler() - TLB invalidation done handler
* @guc: guc
* @msg: message indicating TLB invalidation done
* @len: length of message
*
* Parse seqno of TLB invalidation, wake any waiters for seqno, and signal any
* invalidation fences for seqno. Algorithm for this depends on seqno being
* received in-order and asserts this assumption.
*
* Return: 0 on success, -EPROTO for malformed messages.
*/
int xe_guc_tlb_inval_done_handler(struct xe_guc *guc, u32 *msg, u32 len)
{
struct xe_gt *gt = guc_to_gt(guc);
if (unlikely(len != 1))
return -EPROTO;
xe_tlb_inval_done_handler(&gt->tlb_inval, msg[0]);
return 0;
}

19
drivers/gpu/drm/xe/xe_guc_tlb_inval.h Normal file
View File

@ -0,0 +1,19 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef _XE_GUC_TLB_INVAL_H_
#define _XE_GUC_TLB_INVAL_H_
#include <linux/types.h>
struct xe_guc;
struct xe_tlb_inval;
void xe_guc_tlb_inval_init_early(struct xe_guc *guc,
struct xe_tlb_inval *tlb_inval);
int xe_guc_tlb_inval_done_handler(struct xe_guc *guc, u32 *msg, u32 len);
#endif

6
drivers/gpu/drm/xe/xe_guc_types.h
View File

@ -85,6 +85,12 @@ struct xe_guc {
struct xarray exec_queue_lookup;
/** @submission_state.stopped: submissions are stopped */
atomic_t stopped;
/**
* @submission_state.reset_blocked: reset attempts are blocked;
* blocking reset in order to delay it may be required if running
* an operation which is sensitive to resets.
*/
atomic_t reset_blocked;
/** @submission_state.lock: protects submission state */
struct mutex lock;
/** @submission_state.enabled: submission is enabled */

2
drivers/gpu/drm/xe/xe_heci_gsc.c
View File

@ -197,7 +197,7 @@ int xe_heci_gsc_init(struct xe_device *xe)
if (ret)
return ret;
if (!def->use_polling && !xe_survivability_mode_is_enabled(xe)) {
if (!def->use_polling && !xe_survivability_mode_is_boot_enabled(xe)) {
ret = heci_gsc_irq_setup(xe);
if (ret)
return ret;

2
drivers/gpu/drm/xe/xe_hw_engine.c
View File

@ -576,7 +576,7 @@ static void adjust_idledly(struct xe_hw_engine *hwe)
u32 maxcnt_units_ns = 640;
bool inhibit_switch = 0;
if (!IS_SRIOV_VF(gt_to_xe(hwe->gt)) && XE_WA(gt, 16023105232)) {
if (!IS_SRIOV_VF(gt_to_xe(hwe->gt)) && XE_GT_WA(gt, 16023105232)) {
idledly = xe_mmio_read32(&gt->mmio, RING_IDLEDLY(hwe->mmio_base));
maxcnt = xe_mmio_read32(&gt->mmio, RING_PWRCTX_MAXCNT(hwe->mmio_base));

4
drivers/gpu/drm/xe/xe_hw_engine_group.c
View File

@ -103,8 +103,8 @@ int xe_hw_engine_setup_groups(struct xe_gt *gt)
break;
case XE_ENGINE_CLASS_OTHER:
break;
default:
drm_warn(&xe->drm, "NOT POSSIBLE");
case XE_ENGINE_CLASS_MAX:
xe_gt_assert(gt, false);
}
}

182
drivers/gpu/drm/xe/xe_hw_error.c Normal file
View File

@ -0,0 +1,182 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2025 Intel Corporation
*/
#include <linux/fault-inject.h>
#include "regs/xe_gsc_regs.h"
#include "regs/xe_hw_error_regs.h"
#include "regs/xe_irq_regs.h"
#include "xe_device.h"
#include "xe_hw_error.h"
#include "xe_mmio.h"
#include "xe_survivability_mode.h"
#define HEC_UNCORR_FW_ERR_BITS 4
extern struct fault_attr inject_csc_hw_error;
/* Error categories reported by hardware */
enum hardware_error {
HARDWARE_ERROR_CORRECTABLE = 0,
HARDWARE_ERROR_NONFATAL = 1,
HARDWARE_ERROR_FATAL = 2,
HARDWARE_ERROR_MAX,
};
static const char * const hec_uncorrected_fw_errors[] = {
"Fatal",
"CSE Disabled",
"FD Corruption",
"Data Corruption"
};
static const char *hw_error_to_str(const enum hardware_error hw_err)
{
switch (hw_err) {
case HARDWARE_ERROR_CORRECTABLE:
return "CORRECTABLE";
case HARDWARE_ERROR_NONFATAL:
return "NONFATAL";
case HARDWARE_ERROR_FATAL:
return "FATAL";
default:
return "UNKNOWN";
}
}
static bool fault_inject_csc_hw_error(void)
{
return IS_ENABLED(CONFIG_DEBUG_FS) && should_fail(&inject_csc_hw_error, 1);
}
static void csc_hw_error_work(struct work_struct *work)
{
struct xe_tile *tile = container_of(work, typeof(*tile), csc_hw_error_work);
struct xe_device *xe = tile_to_xe(tile);
int ret;
ret = xe_survivability_mode_runtime_enable(xe);
if (ret)
drm_err(&xe->drm, "Failed to enable runtime survivability mode\n");
}
static void csc_hw_error_handler(struct xe_tile *tile, const enum hardware_error hw_err)
{
const char *hw_err_str = hw_error_to_str(hw_err);
struct xe_device *xe = tile_to_xe(tile);
struct xe_mmio *mmio = &tile->mmio;
u32 base, err_bit, err_src;
unsigned long fw_err;
if (xe->info.platform != XE_BATTLEMAGE)
return;
base = BMG_GSC_HECI1_BASE;
lockdep_assert_held(&xe->irq.lock);
err_src = xe_mmio_read32(mmio, HEC_UNCORR_ERR_STATUS(base));
if (!err_src) {
drm_err_ratelimited(&xe->drm, HW_ERR "Tile%d reported HEC_ERR_STATUS_%s blank\n",
tile->id, hw_err_str);
return;
}
if (err_src & UNCORR_FW_REPORTED_ERR) {
fw_err = xe_mmio_read32(mmio, HEC_UNCORR_FW_ERR_DW0(base));
for_each_set_bit(err_bit, &fw_err, HEC_UNCORR_FW_ERR_BITS) {
drm_err_ratelimited(&xe->drm, HW_ERR
"%s: HEC Uncorrected FW %s error reported, bit[%d] is set\n",
hw_err_str, hec_uncorrected_fw_errors[err_bit],
err_bit);
schedule_work(&tile->csc_hw_error_work);
}
}
xe_mmio_write32(mmio, HEC_UNCORR_ERR_STATUS(base), err_src);
}
static void hw_error_source_handler(struct xe_tile *tile, const enum hardware_error hw_err)
{
const char *hw_err_str = hw_error_to_str(hw_err);
struct xe_device *xe = tile_to_xe(tile);
unsigned long flags;
u32 err_src;
if (xe->info.platform != XE_BATTLEMAGE)
return;
spin_lock_irqsave(&xe->irq.lock, flags);
err_src = xe_mmio_read32(&tile->mmio, DEV_ERR_STAT_REG(hw_err));
if (!err_src) {
drm_err_ratelimited(&xe->drm, HW_ERR "Tile%d reported DEV_ERR_STAT_%s blank!\n",
tile->id, hw_err_str);
goto unlock;
}
if (err_src & XE_CSC_ERROR)
csc_hw_error_handler(tile, hw_err);
xe_mmio_write32(&tile->mmio, DEV_ERR_STAT_REG(hw_err), err_src);
unlock:
spin_unlock_irqrestore(&xe->irq.lock, flags);
}
/**
* xe_hw_error_irq_handler - irq handling for hw errors
* @tile: tile instance
* @master_ctl: value read from master interrupt register
*
* Xe platforms add three error bits to the master interrupt register to support error handling.
* These three bits are used to convey the class of error FATAL, NONFATAL, or CORRECTABLE.
* To process the interrupt, determine the source of error by reading the Device Error Source
* Register that corresponds to the class of error being serviced.
*/
void xe_hw_error_irq_handler(struct xe_tile *tile, const u32 master_ctl)
{
enum hardware_error hw_err;
if (fault_inject_csc_hw_error())
schedule_work(&tile->csc_hw_error_work);
for (hw_err = 0; hw_err < HARDWARE_ERROR_MAX; hw_err++)
if (master_ctl & ERROR_IRQ(hw_err))
hw_error_source_handler(tile, hw_err);
}
/*
* Process hardware errors during boot
*/
static void process_hw_errors(struct xe_device *xe)
{
struct xe_tile *tile;
u32 master_ctl;
u8 id;
for_each_tile(tile, xe, id) {
master_ctl = xe_mmio_read32(&tile->mmio, GFX_MSTR_IRQ);
xe_hw_error_irq_handler(tile, master_ctl);
xe_mmio_write32(&tile->mmio, GFX_MSTR_IRQ, master_ctl);
}
}
/**
* xe_hw_error_init - Initialize hw errors
* @xe: xe device instance
*
* Initialize and check for errors that occurred during boot
* prior to driver load
*/
void xe_hw_error_init(struct xe_device *xe)
{
struct xe_tile *tile = xe_device_get_root_tile(xe);
if (!IS_DGFX(xe) || IS_SRIOV_VF(xe))
return;
INIT_WORK(&tile->csc_hw_error_work, csc_hw_error_work);
process_hw_errors(xe);
}

15
drivers/gpu/drm/xe/xe_hw_error.h Normal file
View File

@ -0,0 +1,15 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef XE_HW_ERROR_H_
#define XE_HW_ERROR_H_
#include <linux/types.h>
struct xe_tile;
struct xe_device;
void xe_hw_error_irq_handler(struct xe_tile *tile, const u32 master_ctl);
void xe_hw_error_init(struct xe_device *xe);
#endif

8
drivers/gpu/drm/xe/xe_hwmon.c
View File

@ -179,7 +179,7 @@ static int xe_hwmon_pcode_rmw_power_limit(const struct xe_hwmon *hwmon, u32 attr
u32 clr, u32 set)
{
struct xe_tile *root_tile = xe_device_get_root_tile(hwmon->xe);
u32 val0, val1;
u32 val0 = 0, val1 = 0;
int ret = 0;
ret = xe_pcode_read(root_tile, PCODE_MBOX(PCODE_POWER_SETUP,
@ -734,7 +734,7 @@ static int xe_hwmon_power_curr_crit_read(struct xe_hwmon *hwmon, int channel,
long *value, u32 scale_factor)
{
int ret;
u32 uval;
u32 uval = 0;
mutex_lock(&hwmon->hwmon_lock);
@ -918,7 +918,7 @@ xe_hwmon_power_write(struct xe_hwmon *hwmon, u32 attr, int channel, long val)
static umode_t
xe_hwmon_curr_is_visible(const struct xe_hwmon *hwmon, u32 attr, int channel)
{
u32 uval;
u32 uval = 0;
/* hwmon sysfs attribute of current available only for package */
if (channel != CHANNEL_PKG)
@ -1020,7 +1020,7 @@ xe_hwmon_energy_read(struct xe_hwmon *hwmon, u32 attr, int channel, long *val)
static umode_t
xe_hwmon_fan_is_visible(struct xe_hwmon *hwmon, u32 attr, int channel)
{
u32 uval;
u32 uval = 0;
if (!hwmon->xe->info.has_fan_control)
return 0;

18
drivers/gpu/drm/xe/xe_i2c.c
View File

@ -146,6 +146,20 @@ static void xe_i2c_unregister_adapter(struct xe_i2c *i2c)
fwnode_remove_software_node(i2c->adapter_node);
}
/**
* xe_i2c_present - I2C controller is present and functional
* @xe: xe device instance
*
* Check whether the I2C controller is present and functioning with valid
* endpoint cookie.
*
* Return: %true if present, %false otherwise.
*/
bool xe_i2c_present(struct xe_device *xe)
{
return xe->i2c && xe->i2c->ep.cookie == XE_I2C_EP_COOKIE_DEVICE;
}
/**
* xe_i2c_irq_handler: Handler for I2C interrupts
* @xe: xe device instance
@ -230,7 +244,7 @@ void xe_i2c_pm_suspend(struct xe_device *xe)
{
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
if (!xe->i2c || xe->i2c->ep.cookie != XE_I2C_EP_COOKIE_DEVICE)
if (!xe_i2c_present(xe))
return;
xe_mmio_rmw32(mmio, I2C_CONFIG_PMCSR, PCI_PM_CTRL_STATE_MASK, (__force u32)PCI_D3hot);
@ -241,7 +255,7 @@ void xe_i2c_pm_resume(struct xe_device *xe, bool d3cold)
{
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
if (!xe->i2c || xe->i2c->ep.cookie != XE_I2C_EP_COOKIE_DEVICE)
if (!xe_i2c_present(xe))
return;
if (d3cold)

2
drivers/gpu/drm/xe/xe_i2c.h
View File

@ -49,11 +49,13 @@ struct xe_i2c {
#if IS_ENABLED(CONFIG_I2C)
int xe_i2c_probe(struct xe_device *xe);
bool xe_i2c_present(struct xe_device *xe);
void xe_i2c_irq_handler(struct xe_device *xe, u32 master_ctl);
void xe_i2c_pm_suspend(struct xe_device *xe);
void xe_i2c_pm_resume(struct xe_device *xe, bool d3cold);
#else
static inline int xe_i2c_probe(struct xe_device *xe) { return 0; }
static inline bool xe_i2c_present(struct xe_device *xe) { return false; }
static inline void xe_i2c_irq_handler(struct xe_device *xe, u32 master_ctl) { }
static inline void xe_i2c_pm_suspend(struct xe_device *xe) { }
static inline void xe_i2c_pm_resume(struct xe_device *xe, bool d3cold) { }

4
drivers/gpu/drm/xe/xe_irq.c
View File

@ -18,6 +18,7 @@
#include "xe_gt.h"
#include "xe_guc.h"
#include "xe_hw_engine.h"
#include "xe_hw_error.h"
#include "xe_i2c.h"
#include "xe_memirq.h"
#include "xe_mmio.h"
@ -468,6 +469,7 @@ static irqreturn_t dg1_irq_handler(int irq, void *arg)
xe_mmio_write32(mmio, GFX_MSTR_IRQ, master_ctl);
gt_irq_handler(tile, master_ctl, intr_dw, identity);
xe_hw_error_irq_handler(tile, master_ctl);
/*
* Display interrupts (including display backlight operations
@ -756,6 +758,8 @@ int xe_irq_install(struct xe_device *xe)
int nvec = 1;
int err;
xe_hw_error_init(xe);
xe_irq_reset(xe);
if (xe_device_has_msix(xe)) {

21
drivers/gpu/drm/xe/xe_lmtt.c
View File

@ -11,7 +11,7 @@
#include "xe_assert.h"
#include "xe_bo.h"
#include "xe_gt_tlb_invalidation.h"
#include "xe_tlb_inval.h"
#include "xe_lmtt.h"
#include "xe_map.h"
#include "xe_mmio.h"
@ -195,14 +195,17 @@ static void lmtt_setup_dir_ptr(struct xe_lmtt *lmtt)
struct xe_tile *tile = lmtt_to_tile(lmtt);
struct xe_device *xe = tile_to_xe(tile);
dma_addr_t offset = xe_bo_main_addr(lmtt->pd->bo, XE_PAGE_SIZE);
struct xe_gt *gt;
u8 id;
lmtt_debug(lmtt, "DIR offset %pad\n", &offset);
lmtt_assert(lmtt, xe_bo_is_vram(lmtt->pd->bo));
lmtt_assert(lmtt, IS_ALIGNED(offset, SZ_64K));
xe_mmio_write32(&tile->mmio,
GRAPHICS_VER(xe) >= 20 ? XE2_LMEM_CFG : LMEM_CFG,
LMEM_EN | REG_FIELD_PREP(LMTT_DIR_PTR, offset / SZ_64K));
for_each_gt_on_tile(gt, tile, id)
xe_mmio_write32(&gt->mmio,
GRAPHICS_VER(xe) >= 20 ? XE2_LMEM_CFG : LMEM_CFG,
LMEM_EN | REG_FIELD_PREP(LMTT_DIR_PTR, offset / SZ_64K));
}
/**
@ -225,8 +228,8 @@ void xe_lmtt_init_hw(struct xe_lmtt *lmtt)
static int lmtt_invalidate_hw(struct xe_lmtt *lmtt)
{
struct xe_gt_tlb_invalidation_fence fences[XE_MAX_GT_PER_TILE];
struct xe_gt_tlb_invalidation_fence *fence = fences;
struct xe_tlb_inval_fence fences[XE_MAX_GT_PER_TILE];
struct xe_tlb_inval_fence *fence = fences;
struct xe_tile *tile = lmtt_to_tile(lmtt);
struct xe_gt *gt;
int result = 0;
@ -234,8 +237,8 @@ static int lmtt_invalidate_hw(struct xe_lmtt *lmtt)
u8 id;
for_each_gt_on_tile(gt, tile, id) {
xe_gt_tlb_invalidation_fence_init(gt, fence, true);
err = xe_gt_tlb_invalidation_all(gt, fence);
xe_tlb_inval_fence_init(&gt->tlb_inval, fence, true);
err = xe_tlb_inval_all(&gt->tlb_inval, fence);
result = result ?: err;
fence++;
}
@ -249,7 +252,7 @@ static int lmtt_invalidate_hw(struct xe_lmtt *lmtt)
*/
fence = fences;
for_each_gt_on_tile(gt, tile, id)
xe_gt_tlb_invalidation_fence_wait(fence++);
xe_tlb_inval_fence_wait(fence++);
return result;
}

174
drivers/gpu/drm/xe/xe_lrc.c
View File

@ -41,7 +41,6 @@
#define LRC_PPHWSP_SIZE SZ_4K
#define LRC_INDIRECT_CTX_BO_SIZE SZ_4K
#define LRC_INDIRECT_RING_STATE_SIZE SZ_4K
#define LRC_WA_BB_SIZE SZ_4K
/*
* Layout of the LRC and associated data allocated as
@ -76,6 +75,11 @@ lrc_to_xe(struct xe_lrc *lrc)
static bool
gt_engine_needs_indirect_ctx(struct xe_gt *gt, enum xe_engine_class class)
{
if (XE_GT_WA(gt, 16010904313) &&
(class == XE_ENGINE_CLASS_RENDER ||
class == XE_ENGINE_CLASS_COMPUTE))
return true;
return false;
}
@ -692,7 +696,13 @@ u32 xe_lrc_regs_offset(struct xe_lrc *lrc)
return xe_lrc_pphwsp_offset(lrc) + LRC_PPHWSP_SIZE;
}
static size_t lrc_reg_size(struct xe_device *xe)
/**
* xe_lrc_reg_size() - Get size of the LRC registers area within queues
* @xe: the &xe_device struct instance
*
* Returns: Size of the LRC registers area for current platform
*/
size_t xe_lrc_reg_size(struct xe_device *xe)
{
if (GRAPHICS_VERx100(xe) >= 1250)
return 96 * sizeof(u32);
@ -702,7 +712,7 @@ static size_t lrc_reg_size(struct xe_device *xe)
size_t xe_lrc_skip_size(struct xe_device *xe)
{
return LRC_PPHWSP_SIZE + lrc_reg_size(xe);
return LRC_PPHWSP_SIZE + xe_lrc_reg_size(xe);
}
static inline u32 __xe_lrc_seqno_offset(struct xe_lrc *lrc)
@ -943,6 +953,47 @@ static void *empty_lrc_data(struct xe_hw_engine *hwe)
return data;
}
/**
* xe_default_lrc_update_memirq_regs_with_address - Re-compute GGTT references in default LRC
* of given engine.
* @hwe: the &xe_hw_engine struct instance
*/
void xe_default_lrc_update_memirq_regs_with_address(struct xe_hw_engine *hwe)
{
struct xe_gt *gt = hwe->gt;
u32 *regs;
if (!gt->default_lrc[hwe->class])
return;
regs = gt->default_lrc[hwe->class] + LRC_PPHWSP_SIZE;
set_memory_based_intr(regs, hwe);
}
/**
* xe_lrc_update_memirq_regs_with_address - Re-compute GGTT references in mem interrupt data
* for given LRC.
* @lrc: the &xe_lrc struct instance
* @hwe: the &xe_hw_engine struct instance
* @regs: scratch buffer to be used as temporary storage
*/
void xe_lrc_update_memirq_regs_with_address(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
u32 *regs)
{
struct xe_gt *gt = hwe->gt;
struct iosys_map map;
size_t regs_len;
if (!xe_device_uses_memirq(gt_to_xe(gt)))
return;
map = __xe_lrc_regs_map(lrc);
regs_len = xe_lrc_reg_size(gt_to_xe(gt));
xe_map_memcpy_from(gt_to_xe(gt), regs, &map, 0, regs_len);
set_memory_based_intr(regs, hwe);
xe_map_memcpy_to(gt_to_xe(gt), &map, 0, regs, regs_len);
}
static void xe_lrc_set_ppgtt(struct xe_lrc *lrc, struct xe_vm *vm)
{
u64 desc = xe_vm_pdp4_descriptor(vm, gt_to_tile(lrc->gt));
@ -1014,6 +1065,63 @@ static ssize_t setup_utilization_wa(struct xe_lrc *lrc,
return cmd - batch;
}
static ssize_t setup_timestamp_wa(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
u32 *batch, size_t max_len)
{
const u32 ts_addr = __xe_lrc_ctx_timestamp_ggtt_addr(lrc);
u32 *cmd = batch;
if (!XE_GT_WA(lrc->gt, 16010904313) ||
!(hwe->class == XE_ENGINE_CLASS_RENDER ||
hwe->class == XE_ENGINE_CLASS_COMPUTE ||
hwe->class == XE_ENGINE_CLASS_COPY ||
hwe->class == XE_ENGINE_CLASS_VIDEO_DECODE ||
hwe->class == XE_ENGINE_CLASS_VIDEO_ENHANCE))
return 0;
if (xe_gt_WARN_ON(lrc->gt, max_len < 12))
return -ENOSPC;
*cmd++ = MI_LOAD_REGISTER_MEM | MI_LRM_USE_GGTT | MI_LRI_LRM_CS_MMIO |
MI_LRM_ASYNC;
*cmd++ = RING_CTX_TIMESTAMP(0).addr;
*cmd++ = ts_addr;
*cmd++ = 0;
*cmd++ = MI_LOAD_REGISTER_MEM | MI_LRM_USE_GGTT | MI_LRI_LRM_CS_MMIO |
MI_LRM_ASYNC;
*cmd++ = RING_CTX_TIMESTAMP(0).addr;
*cmd++ = ts_addr;
*cmd++ = 0;
*cmd++ = MI_LOAD_REGISTER_MEM | MI_LRM_USE_GGTT | MI_LRI_LRM_CS_MMIO;
*cmd++ = RING_CTX_TIMESTAMP(0).addr;
*cmd++ = ts_addr;
*cmd++ = 0;
return cmd - batch;
}
static ssize_t setup_invalidate_state_cache_wa(struct xe_lrc *lrc,
struct xe_hw_engine *hwe,
u32 *batch, size_t max_len)
{
u32 *cmd = batch;
if (!XE_GT_WA(lrc->gt, 18022495364) ||
hwe->class != XE_ENGINE_CLASS_RENDER)
return 0;
if (xe_gt_WARN_ON(lrc->gt, max_len < 3))
return -ENOSPC;
*cmd++ = MI_LOAD_REGISTER_IMM | MI_LRI_NUM_REGS(1);
*cmd++ = CS_DEBUG_MODE1(0).addr;
*cmd++ = _MASKED_BIT_ENABLE(INSTRUCTION_STATE_CACHE_INVALIDATE);
return cmd - batch;
}
struct bo_setup {
ssize_t (*setup)(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
u32 *batch, size_t max_size);
@ -1040,13 +1148,11 @@ static int setup_bo(struct bo_setup_state *state)
ssize_t remain;
if (state->lrc->bo->vmap.is_iomem) {
state->buffer = kmalloc(state->max_size, GFP_KERNEL);
if (!state->buffer)
return -ENOMEM;
state->ptr = state->buffer;
} else {
state->ptr = state->lrc->bo->vmap.vaddr + state->offset;
state->buffer = NULL;
}
remain = state->max_size / sizeof(u32);
@ -1071,7 +1177,6 @@ static int setup_bo(struct bo_setup_state *state)
return 0;
fail:
kfree(state->buffer);
return -ENOSPC;
}
@ -1083,18 +1188,27 @@ static void finish_bo(struct bo_setup_state *state)
xe_map_memcpy_to(gt_to_xe(state->lrc->gt), &state->lrc->bo->vmap,
state->offset, state->buffer,
state->written * sizeof(u32));
kfree(state->buffer);
}
static int setup_wa_bb(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
/**
* xe_lrc_setup_wa_bb_with_scratch - Execute all wa bb setup callbacks.
* @lrc: the &xe_lrc struct instance
* @hwe: the &xe_hw_engine struct instance
* @scratch: preallocated scratch buffer for temporary storage
* Return: 0 on success, negative error code on failure
*/
int xe_lrc_setup_wa_bb_with_scratch(struct xe_lrc *lrc, struct xe_hw_engine *hwe, u32 *scratch)
{
static const struct bo_setup funcs[] = {
{ .setup = setup_timestamp_wa },
{ .setup = setup_invalidate_state_cache_wa },
{ .setup = setup_utilization_wa },
};
struct bo_setup_state state = {
.lrc = lrc,
.hwe = hwe,
.max_size = LRC_WA_BB_SIZE,
.buffer = scratch,
.reserve_dw = 1,
.offset = __xe_lrc_wa_bb_offset(lrc),
.funcs = funcs,
@ -1117,15 +1231,32 @@ static int setup_wa_bb(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
return 0;
}
static int setup_wa_bb(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
{
u32 *buf = NULL;
int ret;
if (lrc->bo->vmap.is_iomem)
buf = kmalloc(LRC_WA_BB_SIZE, GFP_KERNEL);
ret = xe_lrc_setup_wa_bb_with_scratch(lrc, hwe, buf);
kfree(buf);
return ret;
}
static int
setup_indirect_ctx(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
{
static struct bo_setup rcs_funcs[] = {
{ .setup = setup_timestamp_wa },
};
struct bo_setup_state state = {
.lrc = lrc,
.hwe = hwe,
.max_size = (63 * 64) /* max 63 cachelines */,
.buffer = NULL,
.offset = __xe_lrc_indirect_ctx_offset(lrc),
};
int ret;
@ -1142,9 +1273,14 @@ setup_indirect_ctx(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
if (xe_gt_WARN_ON(lrc->gt, !state.funcs))
return 0;
if (lrc->bo->vmap.is_iomem)
state.buffer = kmalloc(state.max_size, GFP_KERNEL);
ret = setup_bo(&state);
if (ret)
if (ret) {
kfree(state.buffer);
return ret;
}
/*
* Align to 64B cacheline so there's no garbage at the end for CS to
@ -1156,6 +1292,7 @@ setup_indirect_ctx(struct xe_lrc *lrc, struct xe_hw_engine *hwe)
}
finish_bo(&state);
kfree(state.buffer);
xe_lrc_write_ctx_reg(lrc,
CTX_CS_INDIRECT_CTX,
@ -1374,6 +1511,23 @@ void xe_lrc_destroy(struct kref *ref)
kfree(lrc);
}
/**
* xe_lrc_update_hwctx_regs_with_address - Re-compute GGTT references within given LRC.
* @lrc: the &xe_lrc struct instance
*/
void xe_lrc_update_hwctx_regs_with_address(struct xe_lrc *lrc)
{
if (xe_lrc_has_indirect_ring_state(lrc)) {
xe_lrc_write_ctx_reg(lrc, CTX_INDIRECT_RING_STATE,
__xe_lrc_indirect_ring_ggtt_addr(lrc));
xe_lrc_write_indirect_ctx_reg(lrc, INDIRECT_CTX_RING_START,
__xe_lrc_ring_ggtt_addr(lrc));
} else {
xe_lrc_write_ctx_reg(lrc, CTX_RING_START, __xe_lrc_ring_ggtt_addr(lrc));
}
}
void xe_lrc_set_ring_tail(struct xe_lrc *lrc, u32 tail)
{
if (xe_lrc_has_indirect_ring_state(lrc))
@ -1939,7 +2093,7 @@ u32 *xe_lrc_emit_hwe_state_instructions(struct xe_exec_queue *q, u32 *cs)
* continue to emit all of the SVG state since it's best not to leak
* any of the state between contexts, even if that leakage is harmless.
*/
if (XE_WA(gt, 14019789679) && q->hwe->class == XE_ENGINE_CLASS_RENDER) {
if (XE_GT_WA(gt, 14019789679) && q->hwe->class == XE_ENGINE_CLASS_RENDER) {
state_table = xe_hpg_svg_state;
state_table_size = ARRAY_SIZE(xe_hpg_svg_state);
}

9
drivers/gpu/drm/xe/xe_lrc.h
View File

@ -42,6 +42,8 @@ struct xe_lrc_snapshot {
#define LRC_PPHWSP_FLUSH_INVAL_SCRATCH_ADDR (0x34 * 4)
#define LRC_PPHWSP_PXP_INVAL_SCRATCH_ADDR (0x40 * 4)
#define LRC_WA_BB_SIZE SZ_4K
#define XE_LRC_CREATE_RUNALONE 0x1
#define XE_LRC_CREATE_PXP 0x2
struct xe_lrc *xe_lrc_create(struct xe_hw_engine *hwe, struct xe_vm *vm,
@ -88,6 +90,10 @@ bool xe_lrc_ring_is_idle(struct xe_lrc *lrc);
u32 xe_lrc_indirect_ring_ggtt_addr(struct xe_lrc *lrc);
u32 xe_lrc_ggtt_addr(struct xe_lrc *lrc);
u32 *xe_lrc_regs(struct xe_lrc *lrc);
void xe_lrc_update_hwctx_regs_with_address(struct xe_lrc *lrc);
void xe_default_lrc_update_memirq_regs_with_address(struct xe_hw_engine *hwe);
void xe_lrc_update_memirq_regs_with_address(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
u32 *regs);
u32 xe_lrc_read_ctx_reg(struct xe_lrc *lrc, int reg_nr);
void xe_lrc_write_ctx_reg(struct xe_lrc *lrc, int reg_nr, u32 val);
@ -106,6 +112,7 @@ s32 xe_lrc_start_seqno(struct xe_lrc *lrc);
u32 xe_lrc_parallel_ggtt_addr(struct xe_lrc *lrc);
struct iosys_map xe_lrc_parallel_map(struct xe_lrc *lrc);
size_t xe_lrc_reg_size(struct xe_device *xe);
size_t xe_lrc_skip_size(struct xe_device *xe);
void xe_lrc_dump_default(struct drm_printer *p,
@ -124,6 +131,8 @@ u32 xe_lrc_ctx_timestamp_udw_ggtt_addr(struct xe_lrc *lrc);
u64 xe_lrc_ctx_timestamp(struct xe_lrc *lrc);
u32 xe_lrc_ctx_job_timestamp_ggtt_addr(struct xe_lrc *lrc);
u32 xe_lrc_ctx_job_timestamp(struct xe_lrc *lrc);
int xe_lrc_setup_wa_bb_with_scratch(struct xe_lrc *lrc, struct xe_hw_engine *hwe,
u32 *scratch);
/**
* xe_lrc_update_timestamp - readout LRC timestamp and update cached value

424
drivers/gpu/drm/xe/xe_migrate.c
View File

@ -9,6 +9,7 @@
#include <linux/sizes.h>
#include <drm/drm_managed.h>
#include <drm/drm_pagemap.h>
#include <drm/ttm/ttm_tt.h>
#include <uapi/drm/xe_drm.h>
@ -30,10 +31,12 @@
#include "xe_mocs.h"
#include "xe_pt.h"
#include "xe_res_cursor.h"
#include "xe_sa.h"
#include "xe_sched_job.h"
#include "xe_sync.h"
#include "xe_trace_bo.h"
#include "xe_vm.h"
#include "xe_vram.h"
/**
* struct xe_migrate - migrate context.
@ -84,19 +87,6 @@ struct xe_migrate {
*/
#define MAX_PTE_PER_SDI 0x1FEU
/**
* xe_tile_migrate_exec_queue() - Get this tile's migrate exec queue.
* @tile: The tile.
*
* Returns the default migrate exec queue of this tile.
*
* Return: The default migrate exec queue
*/
struct xe_exec_queue *xe_tile_migrate_exec_queue(struct xe_tile *tile)
{
return tile->migrate->q;
}
static void xe_migrate_fini(void *arg)
{
struct xe_migrate *m = arg;
@ -130,38 +120,39 @@ static u64 xe_migrate_vram_ofs(struct xe_device *xe, u64 addr, bool is_comp_pte)
u64 identity_offset = IDENTITY_OFFSET;
if (GRAPHICS_VER(xe) >= 20 && is_comp_pte)
identity_offset += DIV_ROUND_UP_ULL(xe->mem.vram.actual_physical_size, SZ_1G);
identity_offset += DIV_ROUND_UP_ULL(xe_vram_region_actual_physical_size
(xe->mem.vram), SZ_1G);
addr -= xe->mem.vram.dpa_base;
addr -= xe_vram_region_dpa_base(xe->mem.vram);
return addr + (identity_offset << xe_pt_shift(2));
}
static void xe_migrate_program_identity(struct xe_device *xe, struct xe_vm *vm, struct xe_bo *bo,
u64 map_ofs, u64 vram_offset, u16 pat_index, u64 pt_2m_ofs)
{
struct xe_vram_region *vram = xe->mem.vram;
resource_size_t dpa_base = xe_vram_region_dpa_base(vram);
u64 pos, ofs, flags;
u64 entry;
/* XXX: Unclear if this should be usable_size? */
u64 vram_limit = xe->mem.vram.actual_physical_size +
xe->mem.vram.dpa_base;
u64 vram_limit = xe_vram_region_actual_physical_size(vram) + dpa_base;
u32 level = 2;
ofs = map_ofs + XE_PAGE_SIZE * level + vram_offset * 8;
flags = vm->pt_ops->pte_encode_addr(xe, 0, pat_index, level,
true, 0);
xe_assert(xe, IS_ALIGNED(xe->mem.vram.usable_size, SZ_2M));
xe_assert(xe, IS_ALIGNED(xe_vram_region_usable_size(vram), SZ_2M));
/*
* Use 1GB pages when possible, last chunk always use 2M
* pages as mixing reserved memory (stolen, WOCPM) with a single
* mapping is not allowed on certain platforms.
*/
for (pos = xe->mem.vram.dpa_base; pos < vram_limit;
for (pos = dpa_base; pos < vram_limit;
pos += SZ_1G, ofs += 8) {
if (pos + SZ_1G >= vram_limit) {
entry = vm->pt_ops->pde_encode_bo(bo, pt_2m_ofs,
pat_index);
entry = vm->pt_ops->pde_encode_bo(bo, pt_2m_ofs);
xe_map_wr(xe, &bo->vmap, ofs, u64, entry);
flags = vm->pt_ops->pte_encode_addr(xe, 0,
@ -215,7 +206,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
/* PT30 & PT31 reserved for 2M identity map */
pt29_ofs = xe_bo_size(bo) - 3 * XE_PAGE_SIZE;
entry = vm->pt_ops->pde_encode_bo(bo, pt29_ofs, pat_index);
entry = vm->pt_ops->pde_encode_bo(bo, pt29_ofs);
xe_pt_write(xe, &vm->pt_root[id]->bo->vmap, 0, entry);
map_ofs = (num_entries - num_setup) * XE_PAGE_SIZE;
@ -283,15 +274,14 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
flags = XE_PDE_64K;
entry = vm->pt_ops->pde_encode_bo(bo, map_ofs + (u64)(level - 1) *
XE_PAGE_SIZE, pat_index);
XE_PAGE_SIZE);
xe_map_wr(xe, &bo->vmap, map_ofs + XE_PAGE_SIZE * level, u64,
entry | flags);
}
/* Write PDE's that point to our BO. */
for (i = 0; i < map_ofs / PAGE_SIZE; i++) {
entry = vm->pt_ops->pde_encode_bo(bo, (u64)i * XE_PAGE_SIZE,
pat_index);
for (i = 0; i < map_ofs / XE_PAGE_SIZE; i++) {
entry = vm->pt_ops->pde_encode_bo(bo, (u64)i * XE_PAGE_SIZE);
xe_map_wr(xe, &bo->vmap, map_ofs + XE_PAGE_SIZE +
(i + 1) * 8, u64, entry);
@ -307,11 +297,11 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
/* Identity map the entire vram at 256GiB offset */
if (IS_DGFX(xe)) {
u64 pt30_ofs = xe_bo_size(bo) - 2 * XE_PAGE_SIZE;
resource_size_t actual_phy_size = xe_vram_region_actual_physical_size(xe->mem.vram);
xe_migrate_program_identity(xe, vm, bo, map_ofs, IDENTITY_OFFSET,
pat_index, pt30_ofs);
xe_assert(xe, xe->mem.vram.actual_physical_size <=
(MAX_NUM_PTE - IDENTITY_OFFSET) * SZ_1G);
xe_assert(xe, actual_phy_size <= (MAX_NUM_PTE - IDENTITY_OFFSET) * SZ_1G);
/*
* Identity map the entire vram for compressed pat_index for xe2+
@ -320,11 +310,11 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
if (GRAPHICS_VER(xe) >= 20 && xe_device_has_flat_ccs(xe)) {
u16 comp_pat_index = xe->pat.idx[XE_CACHE_NONE_COMPRESSION];
u64 vram_offset = IDENTITY_OFFSET +
DIV_ROUND_UP_ULL(xe->mem.vram.actual_physical_size, SZ_1G);
DIV_ROUND_UP_ULL(actual_phy_size, SZ_1G);
u64 pt31_ofs = xe_bo_size(bo) - XE_PAGE_SIZE;
xe_assert(xe, xe->mem.vram.actual_physical_size <= (MAX_NUM_PTE -
IDENTITY_OFFSET - IDENTITY_OFFSET / 2) * SZ_1G);
xe_assert(xe, actual_phy_size <= (MAX_NUM_PTE - IDENTITY_OFFSET -
IDENTITY_OFFSET / 2) * SZ_1G);
xe_migrate_program_identity(xe, vm, bo, map_ofs, vram_offset,
comp_pat_index, pt31_ofs);
}
@ -387,38 +377,47 @@ static bool xe_migrate_needs_ccs_emit(struct xe_device *xe)
}
/**
* xe_migrate_init() - Initialize a migrate context
* @tile: Back-pointer to the tile we're initializing for.
* xe_migrate_alloc - Allocate a migrate struct for a given &xe_tile
* @tile: &xe_tile
*
* Return: Pointer to a migrate context on success. Error pointer on error.
* Allocates a &xe_migrate for a given tile.
*
* Return: &xe_migrate on success, or NULL when out of memory.
*/
struct xe_migrate *xe_migrate_init(struct xe_tile *tile)
struct xe_migrate *xe_migrate_alloc(struct xe_tile *tile)
{
struct xe_device *xe = tile_to_xe(tile);
struct xe_migrate *m = drmm_kzalloc(&tile_to_xe(tile)->drm, sizeof(*m), GFP_KERNEL);
if (m)
m->tile = tile;
return m;
}
/**
* xe_migrate_init() - Initialize a migrate context
* @m: The migration context
*
* Return: 0 if successful, negative error code on failure
*/
int xe_migrate_init(struct xe_migrate *m)
{
struct xe_tile *tile = m->tile;
struct xe_gt *primary_gt = tile->primary_gt;
struct xe_migrate *m;
struct xe_device *xe = tile_to_xe(tile);
struct xe_vm *vm;
int err;
m = devm_kzalloc(xe->drm.dev, sizeof(*m), GFP_KERNEL);
if (!m)
return ERR_PTR(-ENOMEM);
m->tile = tile;
/* Special layout, prepared below.. */
vm = xe_vm_create(xe, XE_VM_FLAG_MIGRATION |
XE_VM_FLAG_SET_TILE_ID(tile));
XE_VM_FLAG_SET_TILE_ID(tile), NULL);
if (IS_ERR(vm))
return ERR_CAST(vm);
return PTR_ERR(vm);
xe_vm_lock(vm, false);
err = xe_migrate_prepare_vm(tile, m, vm);
xe_vm_unlock(vm);
if (err) {
xe_vm_close_and_put(vm);
return ERR_PTR(err);
}
if (err)
goto err_out;
if (xe->info.has_usm) {
struct xe_hw_engine *hwe = xe_gt_hw_engine(primary_gt,
@ -427,8 +426,10 @@ struct xe_migrate *xe_migrate_init(struct xe_tile *tile)
false);
u32 logical_mask = xe_migrate_usm_logical_mask(primary_gt);
if (!hwe || !logical_mask)
return ERR_PTR(-EINVAL);
if (!hwe || !logical_mask) {
err = -EINVAL;
goto err_out;
}
/*
* XXX: Currently only reserving 1 (likely slow) BCS instance on
@ -437,16 +438,18 @@ struct xe_migrate *xe_migrate_init(struct xe_tile *tile)
m->q = xe_exec_queue_create(xe, vm, logical_mask, 1, hwe,
EXEC_QUEUE_FLAG_KERNEL |
EXEC_QUEUE_FLAG_PERMANENT |
EXEC_QUEUE_FLAG_HIGH_PRIORITY, 0);
EXEC_QUEUE_FLAG_HIGH_PRIORITY |
EXEC_QUEUE_FLAG_MIGRATE, 0);
} else {
m->q = xe_exec_queue_create_class(xe, primary_gt, vm,
XE_ENGINE_CLASS_COPY,
EXEC_QUEUE_FLAG_KERNEL |
EXEC_QUEUE_FLAG_PERMANENT, 0);
EXEC_QUEUE_FLAG_PERMANENT |
EXEC_QUEUE_FLAG_MIGRATE, 0);
}
if (IS_ERR(m->q)) {
xe_vm_close_and_put(vm);
return ERR_CAST(m->q);
err = PTR_ERR(m->q);
goto err_out;
}
mutex_init(&m->job_mutex);
@ -456,7 +459,7 @@ struct xe_migrate *xe_migrate_init(struct xe_tile *tile)
err = devm_add_action_or_reset(xe->drm.dev, xe_migrate_fini, m);
if (err)
return ERR_PTR(err);
return err;
if (IS_DGFX(xe)) {
if (xe_migrate_needs_ccs_emit(xe))
@ -471,7 +474,12 @@ struct xe_migrate *xe_migrate_init(struct xe_tile *tile)
(unsigned long long)m->min_chunk_size);
}
return m;
return err;
err_out:
xe_vm_close_and_put(vm);
return err;
}
static u64 max_mem_transfer_per_pass(struct xe_device *xe)
@ -896,7 +904,7 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
goto err;
}
xe_sched_job_add_migrate_flush(job, flush_flags);
xe_sched_job_add_migrate_flush(job, flush_flags | MI_INVALIDATE_TLB);
if (!fence) {
err = xe_sched_job_add_deps(job, src_bo->ttm.base.resv,
DMA_RESV_USAGE_BOOKKEEP);
@ -940,6 +948,167 @@ err_sync:
return fence;
}
/**
* xe_migrate_lrc() - Get the LRC from migrate context.
* @migrate: Migrate context.
*
* Return: Pointer to LRC on success, error on failure
*/
struct xe_lrc *xe_migrate_lrc(struct xe_migrate *migrate)
{
return migrate->q->lrc[0];
}
static int emit_flush_invalidate(struct xe_exec_queue *q, u32 *dw, int i,
u32 flags)
{
struct xe_lrc *lrc = xe_exec_queue_lrc(q);
dw[i++] = MI_FLUSH_DW | MI_INVALIDATE_TLB | MI_FLUSH_DW_OP_STOREDW |
MI_FLUSH_IMM_DW | flags;
dw[i++] = lower_32_bits(xe_lrc_start_seqno_ggtt_addr(lrc)) |
MI_FLUSH_DW_USE_GTT;
dw[i++] = upper_32_bits(xe_lrc_start_seqno_ggtt_addr(lrc));
dw[i++] = MI_NOOP;
dw[i++] = MI_NOOP;
return i;
}
/**
* xe_migrate_ccs_rw_copy() - Copy content of TTM resources.
* @tile: Tile whose migration context to be used.
* @q : Execution to be used along with migration context.
* @src_bo: The buffer object @src is currently bound to.
* @read_write : Creates BB commands for CCS read/write.
*
* Creates batch buffer instructions to copy CCS metadata from CCS pool to
* memory and vice versa.
*
* This function should only be called for IGPU.
*
* Return: 0 if successful, negative error code on failure.
*/
int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
struct xe_bo *src_bo,
enum xe_sriov_vf_ccs_rw_ctxs read_write)
{
bool src_is_pltt = read_write == XE_SRIOV_VF_CCS_READ_CTX;
bool dst_is_pltt = read_write == XE_SRIOV_VF_CCS_WRITE_CTX;
struct ttm_resource *src = src_bo->ttm.resource;
struct xe_migrate *m = tile->migrate;
struct xe_gt *gt = tile->primary_gt;
u32 batch_size, batch_size_allocated;
struct xe_device *xe = gt_to_xe(gt);
struct xe_res_cursor src_it, ccs_it;
u64 size = xe_bo_size(src_bo);
struct xe_bb *bb = NULL;
u64 src_L0, src_L0_ofs;
u32 src_L0_pt;
int err;
xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
&ccs_it);
/* Calculate Batch buffer size */
batch_size = 0;
while (size) {
batch_size += 10; /* Flush + ggtt addr + 2 NOP */
u64 ccs_ofs, ccs_size;
u32 ccs_pt;
u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
src_L0 = min_t(u64, max_mem_transfer_per_pass(xe), size);
batch_size += pte_update_size(m, false, src, &src_it, &src_L0,
&src_L0_ofs, &src_L0_pt, 0, 0,
avail_pts);
ccs_size = xe_device_ccs_bytes(xe, src_L0);
batch_size += pte_update_size(m, 0, NULL, &ccs_it, &ccs_size, &ccs_ofs,
&ccs_pt, 0, avail_pts, avail_pts);
xe_assert(xe, IS_ALIGNED(ccs_it.start, PAGE_SIZE));
/* Add copy commands size here */
batch_size += EMIT_COPY_CCS_DW;
size -= src_L0;
}
bb = xe_bb_ccs_new(gt, batch_size, read_write);
if (IS_ERR(bb)) {
drm_err(&xe->drm, "BB allocation failed.\n");
err = PTR_ERR(bb);
goto err_ret;
}
batch_size_allocated = batch_size;
size = xe_bo_size(src_bo);
batch_size = 0;
/*
* Emit PTE and copy commands here.
* The CCS copy command can only support limited size. If the size to be
* copied is more than the limit, divide copy into chunks. So, calculate
* sizes here again before copy command is emitted.
*/
while (size) {
batch_size += 10; /* Flush + ggtt addr + 2 NOP */
u32 flush_flags = 0;
u64 ccs_ofs, ccs_size;
u32 ccs_pt;
u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
src_L0 = xe_migrate_res_sizes(m, &src_it);
batch_size += pte_update_size(m, false, src, &src_it, &src_L0,
&src_L0_ofs, &src_L0_pt, 0, 0,
avail_pts);
ccs_size = xe_device_ccs_bytes(xe, src_L0);
batch_size += pte_update_size(m, 0, NULL, &ccs_it, &ccs_size, &ccs_ofs,
&ccs_pt, 0, avail_pts, avail_pts);
xe_assert(xe, IS_ALIGNED(ccs_it.start, PAGE_SIZE));
batch_size += EMIT_COPY_CCS_DW;
emit_pte(m, bb, src_L0_pt, false, true, &src_it, src_L0, src);
emit_pte(m, bb, ccs_pt, false, false, &ccs_it, ccs_size, src);
bb->len = emit_flush_invalidate(q, bb->cs, bb->len, flush_flags);
flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs, src_is_pltt,
src_L0_ofs, dst_is_pltt,
src_L0, ccs_ofs, true);
bb->len = emit_flush_invalidate(q, bb->cs, bb->len, flush_flags);
size -= src_L0;
}
xe_assert(xe, (batch_size_allocated == bb->len));
src_bo->bb_ccs[read_write] = bb;
return 0;
err_ret:
return err;
}
/**
* xe_get_migrate_exec_queue() - Get the execution queue from migrate context.
* @migrate: Migrate context.
*
* Return: Pointer to execution queue on success, error on failure
*/
struct xe_exec_queue *xe_migrate_exec_queue(struct xe_migrate *migrate)
{
return migrate->q;
}
static void emit_clear_link_copy(struct xe_gt *gt, struct xe_bb *bb, u64 src_ofs,
u32 size, u32 pitch)
{
@ -1119,11 +1288,13 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,
size -= clear_L0;
/* Preemption is enabled again by the ring ops. */
if (clear_vram && xe_migrate_allow_identity(clear_L0, &src_it))
if (clear_vram && xe_migrate_allow_identity(clear_L0, &src_it)) {
xe_res_next(&src_it, clear_L0);
else
emit_pte(m, bb, clear_L0_pt, clear_vram, clear_only_system_ccs,
&src_it, clear_L0, dst);
} else {
emit_pte(m, bb, clear_L0_pt, clear_vram,
clear_only_system_ccs, &src_it, clear_L0, dst);
flush_flags |= MI_INVALIDATE_TLB;
}
bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
update_idx = bb->len;
@ -1134,7 +1305,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,
if (xe_migrate_needs_ccs_emit(xe)) {
emit_copy_ccs(gt, bb, clear_L0_ofs, true,
m->cleared_mem_ofs, false, clear_L0);
flush_flags = MI_FLUSH_DW_CCS;
flush_flags |= MI_FLUSH_DW_CCS;
}
job = xe_bb_create_migration_job(m->q, bb,
@ -1469,6 +1640,8 @@ next_cmd:
goto err_sa;
}
xe_sched_job_add_migrate_flush(job, MI_INVALIDATE_TLB);
if (ops->pre_commit) {
pt_update->job = job;
err = ops->pre_commit(pt_update);
@ -1571,7 +1744,8 @@ static u32 pte_update_cmd_size(u64 size)
static void build_pt_update_batch_sram(struct xe_migrate *m,
struct xe_bb *bb, u32 pt_offset,
dma_addr_t *sram_addr, u32 size)
struct drm_pagemap_addr *sram_addr,
u32 size)
{
u16 pat_index = tile_to_xe(m->tile)->pat.idx[XE_CACHE_WB];
u32 ptes;
@ -1589,14 +1763,18 @@ static void build_pt_update_batch_sram(struct xe_migrate *m,
ptes -= chunk;
while (chunk--) {
u64 addr = sram_addr[i++] & PAGE_MASK;
u64 addr = sram_addr[i].addr & PAGE_MASK;
xe_tile_assert(m->tile, sram_addr[i].proto ==
DRM_INTERCONNECT_SYSTEM);
xe_tile_assert(m->tile, addr);
addr = m->q->vm->pt_ops->pte_encode_addr(m->tile->xe,
addr, pat_index,
0, false, 0);
bb->cs[bb->len++] = lower_32_bits(addr);
bb->cs[bb->len++] = upper_32_bits(addr);
i++;
}
}
}
@ -1612,7 +1790,8 @@ enum xe_migrate_copy_dir {
static struct dma_fence *xe_migrate_vram(struct xe_migrate *m,
unsigned long len,
unsigned long sram_offset,
dma_addr_t *sram_addr, u64 vram_addr,
struct drm_pagemap_addr *sram_addr,
u64 vram_addr,
const enum xe_migrate_copy_dir dir)
{
struct xe_gt *gt = m->tile->primary_gt;
@ -1628,6 +1807,7 @@ static struct dma_fence *xe_migrate_vram(struct xe_migrate *m,
unsigned int pitch = len >= PAGE_SIZE && !(len & ~PAGE_MASK) ?
PAGE_SIZE : 4;
int err;
unsigned long i, j;
if (drm_WARN_ON(&xe->drm, (len & XE_CACHELINE_MASK) ||
(sram_offset | vram_addr) & XE_CACHELINE_MASK))
@ -1644,6 +1824,24 @@ static struct dma_fence *xe_migrate_vram(struct xe_migrate *m,
return ERR_PTR(err);
}
/*
* If the order of a struct drm_pagemap_addr entry is greater than 0,
* the entry is populated by GPU pagemap but subsequent entries within
* the range of that order are not populated.
* build_pt_update_batch_sram() expects a fully populated array of
* struct drm_pagemap_addr. Ensure this is the case even with higher
* orders.
*/
for (i = 0; i < npages;) {
unsigned int order = sram_addr[i].order;
for (j = 1; j < NR_PAGES(order) && i + j < npages; j++)
if (!sram_addr[i + j].addr)
sram_addr[i + j].addr = sram_addr[i].addr + j * PAGE_SIZE;
i += NR_PAGES(order);
}
build_pt_update_batch_sram(m, bb, pt_slot * XE_PAGE_SIZE,
sram_addr, len + sram_offset);
@ -1669,7 +1867,7 @@ static struct dma_fence *xe_migrate_vram(struct xe_migrate *m,
goto err;
}
xe_sched_job_add_migrate_flush(job, 0);
xe_sched_job_add_migrate_flush(job, MI_INVALIDATE_TLB);
mutex_lock(&m->job_mutex);
xe_sched_job_arm(job);
@ -1694,7 +1892,7 @@ err:
* xe_migrate_to_vram() - Migrate to VRAM
* @m: The migration context.
* @npages: Number of pages to migrate.
* @src_addr: Array of dma addresses (source of migrate)
* @src_addr: Array of DMA information (source of migrate)
* @dst_addr: Device physical address of VRAM (destination of migrate)
*
* Copy from an array dma addresses to a VRAM device physical address
@ -1704,7 +1902,7 @@ err:
*/
struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m,
unsigned long npages,
dma_addr_t *src_addr,
struct drm_pagemap_addr *src_addr,
u64 dst_addr)
{
return xe_migrate_vram(m, npages * PAGE_SIZE, 0, src_addr, dst_addr,
@ -1716,7 +1914,7 @@ struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m,
* @m: The migration context.
* @npages: Number of pages to migrate.
* @src_addr: Device physical address of VRAM (source of migrate)
* @dst_addr: Array of dma addresses (destination of migrate)
* @dst_addr: Array of DMA information (destination of migrate)
*
* Copy from a VRAM device physical address to an array dma addresses
*
@ -1726,61 +1924,65 @@ struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m,
struct dma_fence *xe_migrate_from_vram(struct xe_migrate *m,
unsigned long npages,
u64 src_addr,
dma_addr_t *dst_addr)
struct drm_pagemap_addr *dst_addr)
{
return xe_migrate_vram(m, npages * PAGE_SIZE, 0, dst_addr, src_addr,
XE_MIGRATE_COPY_TO_SRAM);
}
static void xe_migrate_dma_unmap(struct xe_device *xe, dma_addr_t *dma_addr,
static void xe_migrate_dma_unmap(struct xe_device *xe,
struct drm_pagemap_addr *pagemap_addr,
int len, int write)
{
unsigned long i, npages = DIV_ROUND_UP(len, PAGE_SIZE);
for (i = 0; i < npages; ++i) {
if (!dma_addr[i])
if (!pagemap_addr[i].addr)
break;
dma_unmap_page(xe->drm.dev, dma_addr[i], PAGE_SIZE,
dma_unmap_page(xe->drm.dev, pagemap_addr[i].addr, PAGE_SIZE,
write ? DMA_TO_DEVICE : DMA_FROM_DEVICE);
}
kfree(dma_addr);
kfree(pagemap_addr);
}
static dma_addr_t *xe_migrate_dma_map(struct xe_device *xe,
void *buf, int len, int write)
static struct drm_pagemap_addr *xe_migrate_dma_map(struct xe_device *xe,
void *buf, int len,
int write)
{
dma_addr_t *dma_addr;
struct drm_pagemap_addr *pagemap_addr;
unsigned long i, npages = DIV_ROUND_UP(len, PAGE_SIZE);
dma_addr = kcalloc(npages, sizeof(*dma_addr), GFP_KERNEL);
if (!dma_addr)
pagemap_addr = kcalloc(npages, sizeof(*pagemap_addr), GFP_KERNEL);
if (!pagemap_addr)
return ERR_PTR(-ENOMEM);
for (i = 0; i < npages; ++i) {
dma_addr_t addr;
struct page *page;
enum dma_data_direction dir = write ? DMA_TO_DEVICE :
DMA_FROM_DEVICE;
if (is_vmalloc_addr(buf))
page = vmalloc_to_page(buf);
else
page = virt_to_page(buf);
addr = dma_map_page(xe->drm.dev,
page, 0, PAGE_SIZE,
write ? DMA_TO_DEVICE :
DMA_FROM_DEVICE);
addr = dma_map_page(xe->drm.dev, page, 0, PAGE_SIZE, dir);
if (dma_mapping_error(xe->drm.dev, addr))
goto err_fault;
dma_addr[i] = addr;
pagemap_addr[i] =
drm_pagemap_addr_encode(addr,
DRM_INTERCONNECT_SYSTEM,
0, dir);
buf += PAGE_SIZE;
}
return dma_addr;
return pagemap_addr;
err_fault:
xe_migrate_dma_unmap(xe, dma_addr, len, write);
xe_migrate_dma_unmap(xe, pagemap_addr, len, write);
return ERR_PTR(-EFAULT);
}
@ -1809,7 +2011,7 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,
struct xe_device *xe = tile_to_xe(tile);
struct xe_res_cursor cursor;
struct dma_fence *fence = NULL;
dma_addr_t *dma_addr;
struct drm_pagemap_addr *pagemap_addr;
unsigned long page_offset = (unsigned long)buf & ~PAGE_MASK;
int bytes_left = len, current_page = 0;
void *orig_buf = buf;
@ -1869,9 +2071,9 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,
return err;
}
dma_addr = xe_migrate_dma_map(xe, buf, len + page_offset, write);
if (IS_ERR(dma_addr))
return PTR_ERR(dma_addr);
pagemap_addr = xe_migrate_dma_map(xe, buf, len + page_offset, write);
if (IS_ERR(pagemap_addr))
return PTR_ERR(pagemap_addr);
xe_res_first(bo->ttm.resource, offset, xe_bo_size(bo) - offset, &cursor);
@ -1895,7 +2097,7 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,
__fence = xe_migrate_vram(m, current_bytes,
(unsigned long)buf & ~PAGE_MASK,
dma_addr + current_page,
&pagemap_addr[current_page],
vram_addr, write ?
XE_MIGRATE_COPY_TO_VRAM :
XE_MIGRATE_COPY_TO_SRAM);
@ -1923,10 +2125,46 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,
dma_fence_put(fence);
out_err:
xe_migrate_dma_unmap(xe, dma_addr, len + page_offset, write);
xe_migrate_dma_unmap(xe, pagemap_addr, len + page_offset, write);
return IS_ERR(fence) ? PTR_ERR(fence) : 0;
}
/**
* xe_migrate_job_lock() - Lock migrate job lock
* @m: The migration context.
* @q: Queue associated with the operation which requires a lock
*
* Lock the migrate job lock if the queue is a migration queue, otherwise
* assert the VM's dma-resv is held (user queue's have own locking).
*/
void xe_migrate_job_lock(struct xe_migrate *m, struct xe_exec_queue *q)
{
bool is_migrate = q == m->q;
if (is_migrate)
mutex_lock(&m->job_mutex);
else
xe_vm_assert_held(q->vm); /* User queues VM's should be locked */
}
/**
* xe_migrate_job_unlock() - Unlock migrate job lock
* @m: The migration context.
* @q: Queue associated with the operation which requires a lock
*
* Unlock the migrate job lock if the queue is a migration queue, otherwise
* assert the VM's dma-resv is held (user queue's have own locking).
*/
void xe_migrate_job_unlock(struct xe_migrate *m, struct xe_exec_queue *q)
{
bool is_migrate = q == m->q;
if (is_migrate)
mutex_unlock(&m->job_mutex);
else
xe_vm_assert_held(q->vm); /* User queues VM's should be locked */
}
#if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
#include "tests/xe_migrate.c"
#endif

29
drivers/gpu/drm/xe/xe_migrate.h
View File

@ -9,11 +9,13 @@
#include <linux/types.h>
struct dma_fence;
struct drm_pagemap_addr;
struct iosys_map;
struct ttm_resource;
struct xe_bo;
struct xe_gt;
struct xe_tlb_inval_job;
struct xe_exec_queue;
struct xe_migrate;
struct xe_migrate_pt_update;
@ -24,6 +26,8 @@ struct xe_vm;
struct xe_vm_pgtable_update;
struct xe_vma;
enum xe_sriov_vf_ccs_rw_ctxs;
/**
* struct xe_migrate_pt_update_ops - Callbacks for the
* xe_migrate_update_pgtables() function.
@ -89,21 +93,30 @@ struct xe_migrate_pt_update {
struct xe_vma_ops *vops;
/** @job: The job if a GPU page-table update. NULL otherwise */
struct xe_sched_job *job;
/**
* @ijob: The TLB invalidation job for primary GT. NULL otherwise
*/
struct xe_tlb_inval_job *ijob;
/**
* @mjob: The TLB invalidation job for media GT. NULL otherwise
*/
struct xe_tlb_inval_job *mjob;
/** @tile_id: Tile ID of the update */
u8 tile_id;
};
struct xe_migrate *xe_migrate_init(struct xe_tile *tile);
struct xe_migrate *xe_migrate_alloc(struct xe_tile *tile);
int xe_migrate_init(struct xe_migrate *m);
struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m,
unsigned long npages,
dma_addr_t *src_addr,
struct drm_pagemap_addr *src_addr,
u64 dst_addr);
struct dma_fence *xe_migrate_from_vram(struct xe_migrate *m,
unsigned long npages,
u64 src_addr,
dma_addr_t *dst_addr);
struct drm_pagemap_addr *dst_addr);
struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
struct xe_bo *src_bo,
@ -112,6 +125,12 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
struct ttm_resource *dst,
bool copy_only_ccs);
int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
struct xe_bo *src_bo,
enum xe_sriov_vf_ccs_rw_ctxs read_write);
struct xe_lrc *xe_migrate_lrc(struct xe_migrate *migrate);
struct xe_exec_queue *xe_migrate_exec_queue(struct xe_migrate *migrate);
int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,
unsigned long offset, void *buf, int len,
int write);
@ -133,5 +152,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,
void xe_migrate_wait(struct xe_migrate *m);
struct xe_exec_queue *xe_tile_migrate_exec_queue(struct xe_tile *tile);
void xe_migrate_job_lock(struct xe_migrate *m, struct xe_exec_queue *q);
void xe_migrate_job_unlock(struct xe_migrate *m, struct xe_exec_queue *q);
#endif

33
drivers/gpu/drm/xe/xe_mmio.c
View File

@ -58,7 +58,6 @@ static void tiles_fini(void *arg)
static void mmio_multi_tile_setup(struct xe_device *xe, size_t tile_mmio_size)
{
struct xe_tile *tile;
struct xe_gt *gt;
u8 id;
/*
@ -68,38 +67,6 @@ static void mmio_multi_tile_setup(struct xe_device *xe, size_t tile_mmio_size)
if (xe->info.tile_count == 1)
return;
/* Possibly override number of tile based on configuration register */
if (!xe->info.skip_mtcfg) {
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
u8 tile_count, gt_count;
u32 mtcfg;
/*
* Although the per-tile mmio regs are not yet initialized, this
* is fine as it's going to the root tile's mmio, that's
* guaranteed to be initialized earlier in xe_mmio_probe_early()
*/
mtcfg = xe_mmio_read32(mmio, XEHP_MTCFG_ADDR);
tile_count = REG_FIELD_GET(TILE_COUNT, mtcfg) + 1;
if (tile_count < xe->info.tile_count) {
drm_info(&xe->drm, "tile_count: %d, reduced_tile_count %d\n",
xe->info.tile_count, tile_count);
xe->info.tile_count = tile_count;
/*
* We've already setup gt_count according to the full
* tile count. Re-calculate it to only include the GTs
* that belong to the remaining tile(s).
*/
gt_count = 0;
for_each_gt(gt, xe, id)
if (gt->info.id < tile_count * xe->info.max_gt_per_tile)
gt_count++;
xe->info.gt_count = gt_count;
}
}
for_each_remote_tile(tile, xe, id)
xe_mmio_init(&tile->mmio, tile, xe->mmio.regs + id * tile_mmio_size, SZ_4M);
}

226
drivers/gpu/drm/xe/xe_mmio_gem.c Normal file
View File

@ -0,0 +1,226 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2025 Intel Corporation
*/
#include "xe_mmio_gem.h"
#include <drm/drm_drv.h>
#include <drm/drm_gem.h>
#include <drm/drm_managed.h>
#include "xe_device_types.h"
/**
* DOC: Exposing MMIO regions to userspace
*
* In certain cases, the driver may allow userspace to mmap a portion of the hardware registers.
*
* This can be done as follows:
* 1. Call xe_mmio_gem_create() to create a GEM object with an mmap-able fake offset.
* 2. Use xe_mmio_gem_mmap_offset() on the created GEM object to retrieve the fake offset.
* 3. Provide the fake offset to userspace.
* 4. Userspace can call mmap with the fake offset. The length provided to mmap
* must match the size of the GEM object.
* 5. When the region is no longer needed, call xe_mmio_gem_destroy() to release the GEM object.
*
* NOTE: The exposed MMIO region must be page-aligned with regards to its BAR offset and size.
*
* WARNING: Exposing MMIO regions to userspace can have security and stability implications.
* Make sure not to expose any sensitive registers.
*/
static void xe_mmio_gem_free(struct drm_gem_object *);
static int xe_mmio_gem_mmap(struct drm_gem_object *, struct vm_area_struct *);
static vm_fault_t xe_mmio_gem_vm_fault(struct vm_fault *);
struct xe_mmio_gem {
struct drm_gem_object base;
phys_addr_t phys_addr;
};
static const struct vm_operations_struct vm_ops = {
.open = drm_gem_vm_open,
.close = drm_gem_vm_close,
.fault = xe_mmio_gem_vm_fault,
};
static const struct drm_gem_object_funcs xe_mmio_gem_funcs = {
.free = xe_mmio_gem_free,
.mmap = xe_mmio_gem_mmap,
.vm_ops = &vm_ops,
};
static inline struct xe_mmio_gem *to_xe_mmio_gem(struct drm_gem_object *obj)
{
return container_of(obj, struct xe_mmio_gem, base);
}
/**
* xe_mmio_gem_create - Expose an MMIO region to userspace
* @xe: The xe device
* @file: DRM file descriptor
* @phys_addr: Start of the exposed MMIO region
* @size: The size of the exposed MMIO region
*
* This function creates a GEM object that exposes an MMIO region with an mmap-able
* fake offset.
*
* See: "Exposing MMIO regions to userspace"
*/
struct xe_mmio_gem *xe_mmio_gem_create(struct xe_device *xe, struct drm_file *file,
phys_addr_t phys_addr, size_t size)
{
struct xe_mmio_gem *obj;
struct drm_gem_object *base;
int err;
if ((phys_addr % PAGE_SIZE != 0) || (size % PAGE_SIZE != 0))
return ERR_PTR(-EINVAL);
obj = kzalloc(sizeof(*obj), GFP_KERNEL);
if (!obj)
return ERR_PTR(-ENOMEM);
base = &obj->base;
base->funcs = &xe_mmio_gem_funcs;
obj->phys_addr = phys_addr;
drm_gem_private_object_init(&xe->drm, base, size);
err = drm_gem_create_mmap_offset(base);
if (err)
goto free_gem;
err = drm_vma_node_allow(&base->vma_node, file);
if (err)
goto free_gem;
return obj;
free_gem:
xe_mmio_gem_free(base);
return ERR_PTR(err);
}
/**
* xe_mmio_gem_mmap_offset - Return the mmap-able fake offset
* @gem: the GEM object created with xe_mmio_gem_create()
*
* This function returns the mmap-able fake offset allocated during
* xe_mmio_gem_create().
*
* See: "Exposing MMIO regions to userspace"
*/
u64 xe_mmio_gem_mmap_offset(struct xe_mmio_gem *gem)
{
return drm_vma_node_offset_addr(&gem->base.vma_node);
}
static void xe_mmio_gem_free(struct drm_gem_object *base)
{
struct xe_mmio_gem *obj = to_xe_mmio_gem(base);
drm_gem_object_release(base);
kfree(obj);
}
/**
* xe_mmio_gem_destroy - Destroy the GEM object that exposes an MMIO region
* @gem: the GEM object to destroy
*
* This function releases resources associated with the GEM object created by
* xe_mmio_gem_create().
*
* See: "Exposing MMIO regions to userspace"
*/
void xe_mmio_gem_destroy(struct xe_mmio_gem *gem)
{
xe_mmio_gem_free(&gem->base);
}
static int xe_mmio_gem_mmap(struct drm_gem_object *base, struct vm_area_struct *vma)
{
if (vma->vm_end - vma->vm_start != base->size)
return -EINVAL;
if ((vma->vm_flags & VM_SHARED) == 0)
return -EINVAL;
/* Set vm_pgoff (used as a fake buffer offset by DRM) to 0 */
vma->vm_pgoff = 0;
vma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags));
vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP |
VM_DONTCOPY | VM_NORESERVE);
/* Defer actual mapping to the fault handler. */
return 0;
}
static void xe_mmio_gem_release_dummy_page(struct drm_device *dev, void *res)
{
__free_page((struct page *)res);
}
static vm_fault_t xe_mmio_gem_vm_fault_dummy_page(struct vm_area_struct *vma)
{
struct drm_gem_object *base = vma->vm_private_data;
struct drm_device *dev = base->dev;
vm_fault_t ret = VM_FAULT_NOPAGE;
struct page *page;
unsigned long pfn;
unsigned long i;
page = alloc_page(GFP_KERNEL | __GFP_ZERO);
if (!page)
return VM_FAULT_OOM;
if (drmm_add_action_or_reset(dev, xe_mmio_gem_release_dummy_page, page))
return VM_FAULT_OOM;
pfn = page_to_pfn(page);
/* Map the entire VMA to the same dummy page */
for (i = 0; i < base->size; i += PAGE_SIZE) {
unsigned long addr = vma->vm_start + i;
ret = vmf_insert_pfn(vma, addr, pfn);
if (ret & VM_FAULT_ERROR)
break;
}
return ret;
}
static vm_fault_t xe_mmio_gem_vm_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct drm_gem_object *base = vma->vm_private_data;
struct xe_mmio_gem *obj = to_xe_mmio_gem(base);
struct drm_device *dev = base->dev;
vm_fault_t ret = VM_FAULT_NOPAGE;
unsigned long i;
int idx;
if (!drm_dev_enter(dev, &idx)) {
/*
* Provide a dummy page to avoid SIGBUS for events such as hot-unplug.
* This gives the userspace the option to recover instead of crashing.
* It is assumed the userspace will receive the notification via some
* other channel (e.g. drm uevent).
*/
return xe_mmio_gem_vm_fault_dummy_page(vma);
}
for (i = 0; i < base->size; i += PAGE_SIZE) {
unsigned long addr = vma->vm_start + i;
unsigned long phys_addr = obj->phys_addr + i;
ret = vmf_insert_pfn(vma, addr, PHYS_PFN(phys_addr));
if (ret & VM_FAULT_ERROR)
break;
}
drm_dev_exit(idx);
return ret;
}

20
drivers/gpu/drm/xe/xe_mmio_gem.h Normal file
View File

@ -0,0 +1,20 @@
/* SPDX-License-Identifier: MIT */
/*
* Copyright © 2025 Intel Corporation
*/
#ifndef _XE_MMIO_GEM_H_
#define _XE_MMIO_GEM_H_
#include <linux/types.h>
struct drm_file;
struct xe_device;
struct xe_mmio_gem;
struct xe_mmio_gem *xe_mmio_gem_create(struct xe_device *xe, struct drm_file *file,
phys_addr_t phys_addr, size_t size);
u64 xe_mmio_gem_mmap_offset(struct xe_mmio_gem *gem);
void xe_mmio_gem_destroy(struct xe_mmio_gem *gem);
#endif /* _XE_MMIO_GEM_H_ */

29
drivers/gpu/drm/xe/xe_module.c
View File

@ -135,24 +135,17 @@ static const struct init_funcs init_funcs[] = {
},
};
static int __init xe_call_init_func(unsigned int i)
static int __init xe_call_init_func(const struct init_funcs *func)
{
if (WARN_ON(i >= ARRAY_SIZE(init_funcs)))
return 0;
if (!init_funcs[i].init)
return 0;
return init_funcs[i].init();
if (func->init)
return func->init();
return 0;
}
static void xe_call_exit_func(unsigned int i)
static void xe_call_exit_func(const struct init_funcs *func)
{
if (WARN_ON(i >= ARRAY_SIZE(init_funcs)))
return;
if (!init_funcs[i].exit)
return;
init_funcs[i].exit();
if (func->exit)
func->exit();
}
static int __init xe_init(void)
@ -160,10 +153,12 @@ static int __init xe_init(void)
int err, i;
for (i = 0; i < ARRAY_SIZE(init_funcs); i++) {
err = xe_call_init_func(i);
err = xe_call_init_func(init_funcs + i);
if (err) {
pr_info("%s: module_init aborted at %ps %pe\n",
DRIVER_NAME, init_funcs[i].init, ERR_PTR(err));
while (i--)
xe_call_exit_func(i);
xe_call_exit_func(init_funcs + i);
return err;
}
}
@ -176,7 +171,7 @@ static void __exit xe_exit(void)
int i;
for (i = ARRAY_SIZE(init_funcs) - 1; i >= 0; i--)
xe_call_exit_func(i);
xe_call_exit_func(init_funcs + i);
}
module_init(xe_init);

8
drivers/gpu/drm/xe/xe_nvm.c
View File

@ -39,17 +39,17 @@ static void xe_nvm_release_dev(struct device *dev)
static bool xe_nvm_non_posted_erase(struct xe_device *xe)
{
struct xe_gt *gt = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
if (xe->info.platform != XE_BATTLEMAGE)
return false;
return !(xe_mmio_read32(&gt->mmio, XE_REG(GEN12_CNTL_PROTECTED_NVM_REG)) &
return !(xe_mmio_read32(mmio, XE_REG(GEN12_CNTL_PROTECTED_NVM_REG)) &
NVM_NON_POSTED_ERASE_CHICKEN_BIT);
}
static bool xe_nvm_writable_override(struct xe_device *xe)
{
struct xe_gt *gt = xe_root_mmio_gt(xe);
struct xe_mmio *mmio = xe_root_tile_mmio(xe);
bool writable_override;
resource_size_t base;
@ -72,7 +72,7 @@ static bool xe_nvm_writable_override(struct xe_device *xe)
}
writable_override =
!(xe_mmio_read32(&gt->mmio, HECI_FWSTS2(base)) &
!(xe_mmio_read32(mmio, HECI_FWSTS2(base)) &
HECI_FW_STATUS_2_NVM_ACCESS_MODE);
if (writable_override)
drm_info(&xe->drm, "NVM access overridden by jumper\n");

8
drivers/gpu/drm/xe/xe_oa.c
View File

@ -822,7 +822,7 @@ static void xe_oa_disable_metric_set(struct xe_oa_stream *stream)
u32 sqcnt1;
/* Enable thread stall DOP gating and EU DOP gating. */
if (XE_WA(stream->gt, 1508761755)) {
if (XE_GT_WA(stream->gt, 1508761755)) {
xe_gt_mcr_multicast_write(stream->gt, ROW_CHICKEN,
_MASKED_BIT_DISABLE(STALL_DOP_GATING_DISABLE));
xe_gt_mcr_multicast_write(stream->gt, ROW_CHICKEN2,
@ -1079,7 +1079,7 @@ static int xe_oa_enable_metric_set(struct xe_oa_stream *stream)
* EU NOA signals behave incorrectly if EU clock gating is enabled.
* Disable thread stall DOP gating and EU DOP gating.
*/
if (XE_WA(stream->gt, 1508761755)) {
if (XE_GT_WA(stream->gt, 1508761755)) {
xe_gt_mcr_multicast_write(stream->gt, ROW_CHICKEN,
_MASKED_BIT_ENABLE(STALL_DOP_GATING_DISABLE));
xe_gt_mcr_multicast_write(stream->gt, ROW_CHICKEN2,
@ -1754,7 +1754,7 @@ static int xe_oa_stream_init(struct xe_oa_stream *stream,
* GuC reset of engines causes OA to lose configuration
* state. Prevent this by overriding GUCRC mode.
*/
if (XE_WA(stream->gt, 1509372804)) {
if (XE_GT_WA(stream->gt, 1509372804)) {
ret = xe_guc_pc_override_gucrc_mode(&gt->uc.guc.pc,
SLPC_GUCRC_MODE_GUCRC_NO_RC6);
if (ret)
@ -1886,7 +1886,7 @@ u32 xe_oa_timestamp_frequency(struct xe_gt *gt)
{
u32 reg, shift;
if (XE_WA(gt, 18013179988) || XE_WA(gt, 14015568240)) {
if (XE_GT_WA(gt, 18013179988) || XE_GT_WA(gt, 14015568240)) {
xe_pm_runtime_get(gt_to_xe(gt));
reg = xe_mmio_read32(&gt->mmio, RPM_CONFIG0);
xe_pm_runtime_put(gt_to_xe(gt));

76
drivers/gpu/drm/xe/xe_pci.c
View File

@ -17,6 +17,8 @@
#include "display/xe_display.h"
#include "regs/xe_gt_regs.h"
#include "regs/xe_regs.h"
#include "xe_configfs.h"
#include "xe_device.h"
#include "xe_drv.h"
#include "xe_gt.h"
@ -55,7 +57,7 @@ static const struct xe_graphics_desc graphics_xelp = {
};
#define XE_HP_FEATURES \
.has_range_tlb_invalidation = true, \
.has_range_tlb_inval = true, \
.va_bits = 48, \
.vm_max_level = 3
@ -103,7 +105,7 @@ static const struct xe_graphics_desc graphics_xelpg = {
.has_asid = 1, \
.has_atomic_enable_pte_bit = 1, \
.has_flat_ccs = 1, \
.has_range_tlb_invalidation = 1, \
.has_range_tlb_inval = 1, \
.has_usm = 1, \
.has_64bit_timestamp = 1, \
.va_bits = 48, \
@ -169,6 +171,7 @@ static const struct xe_device_desc tgl_desc = {
.dma_mask_size = 39,
.has_display = true,
.has_llc = true,
.has_sriov = true,
.max_gt_per_tile = 1,
.require_force_probe = true,
};
@ -193,6 +196,7 @@ static const struct xe_device_desc adl_s_desc = {
.dma_mask_size = 39,
.has_display = true,
.has_llc = true,
.has_sriov = true,
.max_gt_per_tile = 1,
.require_force_probe = true,
.subplatforms = (const struct xe_subplatform_desc[]) {
@ -210,6 +214,7 @@ static const struct xe_device_desc adl_p_desc = {
.dma_mask_size = 39,
.has_display = true,
.has_llc = true,
.has_sriov = true,
.max_gt_per_tile = 1,
.require_force_probe = true,
.subplatforms = (const struct xe_subplatform_desc[]) {
@ -225,6 +230,7 @@ static const struct xe_device_desc adl_n_desc = {
.dma_mask_size = 39,
.has_display = true,
.has_llc = true,
.has_sriov = true,
.max_gt_per_tile = 1,
.require_force_probe = true,
};
@ -270,6 +276,7 @@ static const struct xe_device_desc ats_m_desc = {
DG2_FEATURES,
.has_display = false,
.has_sriov = true,
};
static const struct xe_device_desc dg2_desc = {
@ -598,6 +605,44 @@ static int xe_info_init_early(struct xe_device *xe,
return 0;
}
/*
* Possibly override number of tile based on configuration register.
*/
static void xe_info_probe_tile_count(struct xe_device *xe)
{
struct xe_mmio *mmio;
u8 tile_count;
u32 mtcfg;
KUNIT_STATIC_STUB_REDIRECT(xe_info_probe_tile_count, xe);
/*
* Probe for tile count only for platforms that support multiple
* tiles.
*/
if (xe->info.tile_count == 1)
return;
if (xe->info.skip_mtcfg)
return;
mmio = xe_root_tile_mmio(xe);
/*
* Although the per-tile mmio regs are not yet initialized, this
* is fine as it's going to the root tile's mmio, that's
* guaranteed to be initialized earlier in xe_mmio_probe_early()
*/
mtcfg = xe_mmio_read32(mmio, XEHP_MTCFG_ADDR);
tile_count = REG_FIELD_GET(TILE_COUNT, mtcfg) + 1;
if (tile_count < xe->info.tile_count) {
drm_info(&xe->drm, "tile_count: %d, reduced_tile_count %d\n",
xe->info.tile_count, tile_count);
xe->info.tile_count = tile_count;
}
}
/*
* Initialize device info content that does require knowledge about
* graphics / media IP version.
@ -668,10 +713,12 @@ static int xe_info_init(struct xe_device *xe,
/* Runtime detection may change this later */
xe->info.has_flat_ccs = graphics_desc->has_flat_ccs;
xe->info.has_range_tlb_invalidation = graphics_desc->has_range_tlb_invalidation;
xe->info.has_range_tlb_inval = graphics_desc->has_range_tlb_inval;
xe->info.has_usm = graphics_desc->has_usm;
xe->info.has_64bit_timestamp = graphics_desc->has_64bit_timestamp;
xe_info_probe_tile_count(xe);
for_each_remote_tile(tile, xe, id) {
int err;
@ -687,12 +734,17 @@ static int xe_info_init(struct xe_device *xe,
* All of these together determine the overall GT count.
*/
for_each_tile(tile, xe, id) {
int err;
gt = tile->primary_gt;
gt->info.type = XE_GT_TYPE_MAIN;
gt->info.id = tile->id * xe->info.max_gt_per_tile;
gt->info.has_indirect_ring_state = graphics_desc->has_indirect_ring_state;
gt->info.engine_mask = graphics_desc->hw_engine_mask;
xe->info.gt_count++;
err = xe_tile_alloc_vram(tile);
if (err)
return err;
if (MEDIA_VER(xe) < 13 && media_desc)
gt->info.engine_mask |= media_desc->hw_engine_mask;
@ -713,9 +765,15 @@ static int xe_info_init(struct xe_device *xe,
gt->info.id = tile->id * xe->info.max_gt_per_tile + 1;
gt->info.has_indirect_ring_state = media_desc->has_indirect_ring_state;
gt->info.engine_mask = media_desc->hw_engine_mask;
xe->info.gt_count++;
}
/*
* Now that we have tiles and GTs defined, let's loop over valid GTs
* in order to define gt_count.
*/
for_each_gt(gt, xe, id)
xe->info.gt_count++;
return 0;
}
@ -726,7 +784,7 @@ static void xe_pci_remove(struct pci_dev *pdev)
if (IS_SRIOV_PF(xe))
xe_pci_sriov_configure(pdev, 0);
if (xe_survivability_mode_is_enabled(xe))
if (xe_survivability_mode_is_boot_enabled(xe))
return;
xe_device_remove(xe);
@ -759,6 +817,8 @@ static int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
struct xe_device *xe;
int err;
xe_configfs_check_device(pdev);
if (desc->require_force_probe && !id_forced(pdev->device)) {
dev_info(&pdev->dev,
"Your graphics device %04x is not officially supported\n"
@ -806,7 +866,7 @@ static int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
* flashed through mei. Return success, if survivability mode
* is enabled due to pcode failure or configfs being set
*/
if (xe_survivability_mode_is_enabled(xe))
if (xe_survivability_mode_is_boot_enabled(xe))
return 0;
if (err)
@ -900,7 +960,7 @@ static int xe_pci_suspend(struct device *dev)
struct xe_device *xe = pdev_to_xe_device(pdev);
int err;
if (xe_survivability_mode_is_enabled(xe))
if (xe_survivability_mode_is_boot_enabled(xe))
return -EBUSY;
err = xe_pm_suspend(xe);

2
drivers/gpu/drm/xe/xe_pci_types.h
View File

@ -60,7 +60,7 @@ struct xe_graphics_desc {
u8 has_atomic_enable_pte_bit:1;
u8 has_flat_ccs:1;
u8 has_indirect_ring_state:1;
u8 has_range_tlb_invalidation:1;
u8 has_range_tlb_inval:1;
u8 has_usm:1;
u8 has_64bit_timestamp:1;
};

23
drivers/gpu/drm/xe/xe_pm.c
View File

@ -18,11 +18,12 @@
#include "xe_device.h"
#include "xe_ggtt.h"
#include "xe_gt.h"
#include "xe_guc.h"
#include "xe_gt_idle.h"
#include "xe_i2c.h"
#include "xe_irq.h"
#include "xe_pcode.h"
#include "xe_pxp.h"
#include "xe_sriov_vf_ccs.h"
#include "xe_trace.h"
#include "xe_wa.h"
@ -176,6 +177,9 @@ int xe_pm_resume(struct xe_device *xe)
drm_dbg(&xe->drm, "Resuming device\n");
trace_xe_pm_resume(xe, __builtin_return_address(0));
for_each_gt(gt, xe, id)
xe_gt_idle_disable_c6(gt);
for_each_tile(tile, xe, id)
xe_wa_apply_tile_workarounds(tile);
@ -208,6 +212,9 @@ int xe_pm_resume(struct xe_device *xe)
xe_pxp_pm_resume(xe->pxp);
if (IS_SRIOV_VF(xe))
xe_sriov_vf_ccs_register_context(xe);
drm_dbg(&xe->drm, "Device resumed\n");
return 0;
err:
@ -243,6 +250,10 @@ static void xe_pm_runtime_init(struct xe_device *xe)
{
struct device *dev = xe->drm.dev;
/* Our current VFs do not support RPM. so, disable it */
if (IS_SRIOV_VF(xe))
return;
/*
* Disable the system suspend direct complete optimization.
* We need to ensure that the regular device suspend/resume functions
@ -367,6 +378,10 @@ static void xe_pm_runtime_fini(struct xe_device *xe)
{
struct device *dev = xe->drm.dev;
/* Our current VFs do not support RPM. so, disable it */
if (IS_SRIOV_VF(xe))
return;
pm_runtime_get_sync(dev);
pm_runtime_forbid(dev);
}
@ -525,6 +540,9 @@ int xe_pm_runtime_resume(struct xe_device *xe)
xe_rpm_lockmap_acquire(xe);
for_each_gt(gt, xe, id)
xe_gt_idle_disable_c6(gt);
if (xe->d3cold.allowed) {
err = xe_pcode_ready(xe, true);
if (err)
@ -558,6 +576,9 @@ int xe_pm_runtime_resume(struct xe_device *xe)
xe_pxp_pm_resume(xe->pxp);
if (IS_SRIOV_VF(xe))
xe_sriov_vf_ccs_register_context(xe);
out:
xe_rpm_lockmap_release(xe);
xe_pm_write_callback_task(xe, NULL);

306
drivers/gpu/drm/xe/xe_psmi.c Normal file
View File

@ -0,0 +1,306 @@
// SPDX-License-Identifier: MIT
/*
* Copyright © 2025 Intel Corporation
*/
#include <linux/debugfs.h>
#include "xe_bo.h"
#include "xe_device.h"
#include "xe_configfs.h"
#include "xe_psmi.h"
/*
* PSMI capture support
*
* Requirement for PSMI capture is to have a physically contiguous buffer. The
* PSMI tool owns doing all necessary configuration (MMIO register writes are
* done from user-space). However, KMD needs to provide the PSMI tool with the
* required physical address of the base of PSMI buffer in case of VRAM.
*
* VRAM backed PSMI buffer:
* Buffer is allocated as GEM object and with XE_BO_CREATE_PINNED_BIT flag which
* creates a contiguous allocation. The physical address is returned from
* psmi_debugfs_capture_addr_show(). PSMI tool can mmap the buffer via the
* PCIBAR through sysfs.
*
* SYSTEM memory backed PSMI buffer:
* Interface here does not support allocating from SYSTEM memory region. The
* PSMI tool needs to allocate memory themselves using hugetlbfs. In order to
* get the physical address, user-space can query /proc/[pid]/pagemap. As an
* alternative, CMA debugfs could also be used to allocate reserved CMA memory.
*/
static bool psmi_enabled(struct xe_device *xe)
{
return xe_configfs_get_psmi_enabled(to_pci_dev(xe->drm.dev));
}
static void psmi_free_object(struct xe_bo *bo)
{
xe_bo_lock(bo, NULL);
xe_bo_unpin(bo);
xe_bo_unlock(bo);
xe_bo_put(bo);
}
/*
* Free PSMI capture buffer objects.
*/
static void psmi_cleanup(struct xe_device *xe)
{
unsigned long id, region_mask = xe->psmi.region_mask;
struct xe_bo *bo;
for_each_set_bit(id, &region_mask,
ARRAY_SIZE(xe->psmi.capture_obj)) {
/* smem should never be set */
xe_assert(xe, id);
bo = xe->psmi.capture_obj[id];
if (bo) {
psmi_free_object(bo);
xe->psmi.capture_obj[id] = NULL;
}
}
}
static struct xe_bo *psmi_alloc_object(struct xe_device *xe,
unsigned int id, size_t bo_size)
{
struct xe_bo *bo = NULL;
struct xe_tile *tile;
int err;
if (!id || !bo_size)
return NULL;
tile = &xe->tiles[id - 1];
/* VRAM: Allocate GEM object for the capture buffer */
bo = xe_bo_create_locked(xe, tile, NULL, bo_size,
ttm_bo_type_kernel,
XE_BO_FLAG_VRAM_IF_DGFX(tile) |
XE_BO_FLAG_PINNED |
XE_BO_FLAG_PINNED_LATE_RESTORE |
XE_BO_FLAG_NEEDS_CPU_ACCESS);
if (!IS_ERR(bo)) {
/* Buffer written by HW, ensure stays resident */
err = xe_bo_pin(bo);
if (err)
bo = ERR_PTR(err);
xe_bo_unlock(bo);
}
return bo;
}
/*
* Allocate PSMI capture buffer objects (via debugfs set function), based on
* which regions the user has selected in region_mask. @size: size in bytes
* (should be power of 2)
*
* Always release/free the current buffer objects before attempting to allocate
* new ones. Size == 0 will free all current buffers.
*
* Note, we don't write any registers as the capture tool is already configuring
* all PSMI registers itself via mmio space.
*/
static int psmi_resize_object(struct xe_device *xe, size_t size)
{
unsigned long id, region_mask = xe->psmi.region_mask;
struct xe_bo *bo = NULL;
int err = 0;
/* if resizing, free currently allocated buffers first */
psmi_cleanup(xe);
/* can set size to 0, in which case, now done */
if (!size)
return 0;
for_each_set_bit(id, &region_mask,
ARRAY_SIZE(xe->psmi.capture_obj)) {
/* smem should never be set */
xe_assert(xe, id);
bo = psmi_alloc_object(xe, id, size);
if (IS_ERR(bo)) {
err = PTR_ERR(bo);
break;
}
xe->psmi.capture_obj[id] = bo;
drm_info(&xe->drm,
"PSMI capture size requested: %zu bytes, allocated: %lu:%zu\n",
size, id, bo ? xe_bo_size(bo) : 0);
}
/* on error, reverse what was allocated */
if (err)
psmi_cleanup(xe);
return err;
}
/*
* Returns an address for the capture tool to use to find start of capture
* buffer. Capture tool requires the capability to have a buffer allocated per
* each tile (VRAM region), thus we return an address for each region.
*/
static int psmi_debugfs_capture_addr_show(struct seq_file *m, void *data)
{
struct xe_device *xe = m->private;
unsigned long id, region_mask;
struct xe_bo *bo;
u64 val;
region_mask = xe->psmi.region_mask;
for_each_set_bit(id, &region_mask,
ARRAY_SIZE(xe->psmi.capture_obj)) {
/* smem should never be set */
xe_assert(xe, id);
/* VRAM region */
bo = xe->psmi.capture_obj[id];
if (!bo)
continue;
/* pinned, so don't need bo_lock */
val = __xe_bo_addr(bo, 0, PAGE_SIZE);
seq_printf(m, "%ld: 0x%llx\n", id, val);
}
return 0;
}
/*
* Return capture buffer size, using the size from first allocated object that
* is found. This works because all objects must be of the same size.
*/
static int psmi_debugfs_capture_size_get(void *data, u64 *val)
{
unsigned long id, region_mask;
struct xe_device *xe = data;
struct xe_bo *bo;
region_mask = xe->psmi.region_mask;
for_each_set_bit(id, &region_mask,
ARRAY_SIZE(xe->psmi.capture_obj)) {
/* smem should never be set */
xe_assert(xe, id);
bo = xe->psmi.capture_obj[id];
if (bo) {
*val = xe_bo_size(bo);
return 0;
}
}
/* no capture objects are allocated */
*val = 0;
return 0;
}
/*
* Set size of PSMI capture buffer. This triggers the allocation of capture
* buffer in each memory region as specified with prior write to
* psmi_capture_region_mask.
*/
static int psmi_debugfs_capture_size_set(void *data, u64 val)
{
struct xe_device *xe = data;
/* user must have specified at least one region */
if (!xe->psmi.region_mask)
return -EINVAL;
return psmi_resize_object(xe, val);
}
static int psmi_debugfs_capture_region_mask_get(void *data, u64 *val)
{
struct xe_device *xe = data;
*val = xe->psmi.region_mask;
return 0;
}
/*
* Select VRAM regions for multi-tile devices, only allowed when buffer is not
* currently allocated.
*/
static int psmi_debugfs_capture_region_mask_set(void *data, u64 region_mask)
{
struct xe_device *xe = data;
u64 size = 0;
/* SMEM is not supported (see comments at top of file) */
if (region_mask & 0x1)
return -EOPNOTSUPP;
/* input bitmask should contain only valid TTM regions */
if (!region_mask || region_mask & ~xe->info.mem_region_mask)
return -EINVAL;
/* only allow setting mask if buffer is not yet allocated */
psmi_debugfs_capture_size_get(xe, &size);
if (size)
return -EBUSY;
xe->psmi.region_mask = region_mask;
return 0;
}
DEFINE_SHOW_ATTRIBUTE(psmi_debugfs_capture_addr);
DEFINE_DEBUGFS_ATTRIBUTE(psmi_debugfs_capture_region_mask_fops,
psmi_debugfs_capture_region_mask_get,
psmi_debugfs_capture_region_mask_set,
"0x%llx\n");
DEFINE_DEBUGFS_ATTRIBUTE(psmi_debugfs_capture_size_fops,
psmi_debugfs_capture_size_get,
psmi_debugfs_capture_size_set,
"%lld\n");
void xe_psmi_debugfs_register(struct xe_device *xe)
{
struct drm_minor *minor;
if (!psmi_enabled(xe))
return;
minor = xe->drm.primary;
if (!minor->debugfs_root)
return;
debugfs_create_file("psmi_capture_addr",
0400, minor->debugfs_root, xe,
&psmi_debugfs_capture_addr_fops);
debugfs_create_file("psmi_capture_region_mask",
0600, minor->debugfs_root, xe,
&psmi_debugfs_capture_region_mask_fops);
debugfs_create_file("psmi_capture_size",
0600, minor->debugfs_root, xe,
&psmi_debugfs_capture_size_fops);
}
static void psmi_fini(void *arg)
{
psmi_cleanup(arg);
}
int xe_psmi_init(struct xe_device *xe)
{
if (!psmi_enabled(xe))
return 0;
return devm_add_action(xe->drm.dev, psmi_fini, xe);
}

Some files were not shown because too many files have changed in this diff Show More