From 79f3a0c13dcca7b67fa031e9cf94b85241792143 Mon Sep 17 00:00:00 2001 From: Rafael Aquini Date: Fri, 27 Sep 2024 10:43:35 -0400 Subject: [PATCH] mm/mmap: move vma operations to mm_struct out of the critical section of file mapping lock JIRA: https://issues.redhat.com/browse/RHEL-27743 Conflicts: * merge conflict due to out-of-order backport for upstream commit ad9f006351c3 ("mm: always lock new vma before inserting into vma tree") (rhel commit 4263d177b853) wrongly folding part of the first hunk of this patch. This patch is a backport of the following upstream commit: commit 6852c46c783d20a4c0153d14d2990040e5e6e47e Author: Yu Ma Date: Wed Jul 12 10:57:39 2023 -0400 mm/mmap: move vma operations to mm_struct out of the critical section of file mapping lock UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot osq_lock is observed from do_mmap and exit_mmap. Both of them come from load_elf_binary through the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will call mmap_region to create vma node, initialize it and insert it to vma maintain structure in mm_struct and i_mmap tree of the mapping file, then increase map_count to record the number of vma nodes used. The hot osq_lock is to protect operations on file's i_mmap tree. For the mm_struct member change like vma insertion and map_count update, they do not affect i_mmap tree. Move those operations out of the lock's critical section, to reduce hold time on the lock. With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.5-rc1 due to regression of this benchmark from this commit f1a7941243c102a44e8847e3b94ff4ff3ec56f25 (mm: convert mm's rss stats into percpu_counter). Related discussion and conclusion can be referred at the mail thread initiated by 0day as below: Link: https://lore.kernel.org/linux-mm/a4aa2e13-7187-600b-c628-7e8fb108def0@intel.com/ Link: https://lkml.kernel.org/r/20230712145739.604215-1-yu.ma@intel.com Signed-off-by: Yu Ma Reviewed-by: Tim Chen Cc: Dan Williams Cc: Dave Hansen Cc: Kirill A . Shutemov Cc: Liam R. Howlett Cc: Shakeel Butt Cc: Zhu, Lipeng Signed-off-by: Andrew Morton Signed-off-by: Rafael Aquini --- mm/mmap.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index ef17794ad12e..da27ef6de32d 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -419,10 +419,6 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) if (vma->vm_file) { mapping = vma->vm_file->f_mapping; i_mmap_lock_write(mapping); - } - - - if (mapping) { __vma_link_file(vma, mapping); i_mmap_unlock_write(mapping); } @@ -2849,12 +2845,10 @@ cannot_expand: /* Lock the VMA since it is modified after insertion into VMA tree */ vma_start_write(vma); - if (vma->vm_file) - i_mmap_lock_write(vma->vm_file->f_mapping); - vma_iter_store(&vmi, vma); mm->map_count++; if (vma->vm_file) { + i_mmap_lock_write(vma->vm_file->f_mapping); if (vma->vm_flags & VM_SHARED) mapping_allow_writable(vma->vm_file->f_mapping);