Commit Graph

54 Commits

Author SHA1 Message Date
Artem Savkov ade3f4aa53 bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU.
Bugzilla: https://bugzilla.redhat.com/2166911

commit 8d5a8011b35d387c490a5c977b1d9eb4798aa071
Author: Alexei Starovoitov <ast@kernel.org>
Date:   Fri Sep 2 14:10:51 2022 -0700

    bpf: Batch call_rcu callbacks instead of SLAB_TYPESAFE_BY_RCU.
    
    SLAB_TYPESAFE_BY_RCU makes kmem_caches non mergeable and slows down
    kmem_cache_destroy. All bpf_mem_cache are safe to share across different maps
    and programs. Convert SLAB_TYPESAFE_BY_RCU to batched call_rcu. This change
    solves the memory consumption issue, avoids kmem_cache_destroy latency and
    keeps bpf hash map performance the same.
    
    Signed-off-by: Alexei Starovoitov <ast@kernel.org>
    Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
    Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
    Acked-by: Andrii Nakryiko <andrii@kernel.org>
    Link: https://lore.kernel.org/bpf/20220902211058.60789-10-alexei.starovoitov@gmail.com

Signed-off-by: Artem Savkov <asavkov@redhat.com>
2023-03-06 14:54:07 +01:00
Artem Savkov 9f80443429 bpf: Adjust low/high watermarks in bpf_mem_cache
Bugzilla: https://bugzilla.redhat.com/2166911

commit 7c266178aa51dd2d4fda1312c5990a8a82c83d70
Author: Alexei Starovoitov <ast@kernel.org>
Date:   Fri Sep 2 14:10:50 2022 -0700

    bpf: Adjust low/high watermarks in bpf_mem_cache
    
    The same low/high watermarks for every bucket in bpf_mem_cache consume
    significant amount of memory. Preallocating 64 elements of 4096 bytes each in
    the free list is not efficient. Make low/high watermarks and batching value
    dependent on element size. This change brings significant memory savings.
    
    Signed-off-by: Alexei Starovoitov <ast@kernel.org>
    Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
    Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
    Acked-by: Andrii Nakryiko <andrii@kernel.org>
    Link: https://lore.kernel.org/bpf/20220902211058.60789-9-alexei.starovoitov@gmail.com

Signed-off-by: Artem Savkov <asavkov@redhat.com>
2023-03-06 14:54:07 +01:00
Artem Savkov 8afeb1ac37 bpf: Optimize call_rcu in non-preallocated hash map.
Bugzilla: https://bugzilla.redhat.com/2166911

commit 0fd7c5d43339b783ee3301a05f925d1e52ac87c9
Author: Alexei Starovoitov <ast@kernel.org>
Date:   Fri Sep 2 14:10:49 2022 -0700

    bpf: Optimize call_rcu in non-preallocated hash map.
    
    Doing call_rcu() million times a second becomes a bottle neck.
    Convert non-preallocated hash map from call_rcu to SLAB_TYPESAFE_BY_RCU.
    The rcu critical section is no longer observed for one htab element
    which makes non-preallocated hash map behave just like preallocated hash map.
    The map elements are released back to kernel memory after observing
    rcu critical section.
    This improves 'map_perf_test 4' performance from 100k events per second
    to 250k events per second.
    
    bpf_mem_alloc + percpu_counter + typesafe_by_rcu provide 10x performance
    boost to non-preallocated hash map and make it within few % of preallocated map
    while consuming fraction of memory.
    
    Signed-off-by: Alexei Starovoitov <ast@kernel.org>
    Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
    Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
    Acked-by: Andrii Nakryiko <andrii@kernel.org>
    Link: https://lore.kernel.org/bpf/20220902211058.60789-8-alexei.starovoitov@gmail.com

Signed-off-by: Artem Savkov <asavkov@redhat.com>
2023-03-06 14:54:07 +01:00
Artem Savkov 6ed0429790 bpf: Introduce any context BPF specific memory allocator.
Bugzilla: https://bugzilla.redhat.com/2166911

commit 7c8199e24fa09d2344ae0204527d55d7803e8409
Author: Alexei Starovoitov <ast@kernel.org>
Date:   Fri Sep 2 14:10:43 2022 -0700

    bpf: Introduce any context BPF specific memory allocator.
    
    Tracing BPF programs can attach to kprobe and fentry. Hence they
    run in unknown context where calling plain kmalloc() might not be safe.
    
    Front-end kmalloc() with minimal per-cpu cache of free elements.
    Refill this cache asynchronously from irq_work.
    
    BPF programs always run with migration disabled.
    It's safe to allocate from cache of the current cpu with irqs disabled.
    Free-ing is always done into bucket of the current cpu as well.
    irq_work trims extra free elements from buckets with kfree
    and refills them with kmalloc, so global kmalloc logic takes care
    of freeing objects allocated by one cpu and freed on another.
    
    struct bpf_mem_alloc supports two modes:
    - When size != 0 create kmem_cache and bpf_mem_cache for each cpu.
      This is typical bpf hash map use case when all elements have equal size.
    - When size == 0 allocate 11 bpf_mem_cache-s for each cpu, then rely on
      kmalloc/kfree. Max allocation size is 4096 in this case.
      This is bpf_dynptr and bpf_kptr use case.
    
    bpf_mem_alloc/bpf_mem_free are bpf specific 'wrappers' of kmalloc/kfree.
    bpf_mem_cache_alloc/bpf_mem_cache_free are 'wrappers' of kmem_cache_alloc/kmem_cache_free.
    
    The allocators are NMI-safe from bpf programs only. They are not NMI-safe in general.
    
    Signed-off-by: Alexei Starovoitov <ast@kernel.org>
    Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
    Acked-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
    Acked-by: Andrii Nakryiko <andrii@kernel.org>
    Link: https://lore.kernel.org/bpf/20220902211058.60789-2-alexei.starovoitov@gmail.com

Signed-off-by: Artem Savkov <asavkov@redhat.com>
2023-03-06 14:54:06 +01:00