JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 47b59f3603d49d4898aafb256405b16ccdc68cc7
Author: Nathan Chancellor <nathan@kernel.org>
Date: Wed Apr 24 10:24:09 2024 -0700
selftests/resctrl: ksft_exit_skip() does not return
After commit f7d5bcd35d42 ("selftests: kselftest: Mark functions that
unconditionally call exit() as __noreturn"), ksft_exit_...() functions
are marked as __noreturn, which means the return type should not be
'int' but 'void' because they are not returning anything (and never were
since exit() has always been called).
To facilitate updating the return type of these functions, remove
'return' before the calls to ksft_exit_skip(), as __noreturn prevents
the compiler from warning that a caller of ksft_exit_skip() does not
return a value because the program will terminate upon calling these
functions.
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 6cd368982cf3f972e5298025af0a9d2b69045cfe
Author: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
Date: Tue Feb 27 08:21:43 2024 +0100
selftests/resctrl: Move cleanups out of individual tests
Every test calls its cleanup function at the end of it's test function.
After the cleanup function pointer is added to the test framework this
can be simplified to executing the callback function at the end of the
generic test running function.
Make test cleanup functions static and call them from the end of
run_single_test() from the resctrl_test's cleanup function pointer.
Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit e6487230e952cfd4070c61141f011608841f36eb
Author: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
Date: Tue Feb 27 08:21:42 2024 +0100
selftests/resctrl: Simplify cleanup in ctrl-c handler
Ctrl-c handler isn't aware of what test is currently running. Because of
that it executes all cleanups even if they aren't necessary. Since the
ctrl-c handler uses the sa_sigaction system no parameters can be passed
to it as function arguments.
Add a global variable to make ctrl-c handler aware of the currently run
test and only execute the correct cleanup callback.
Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit ae638551ab64a216364a63c4ee5061f82702fde2
Author: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
Date: Fri Feb 16 09:35:52 2024 +0100
selftests/resctrl: Add non-contiguous CBMs CAT test
Add tests for both L2 and L3 CAT to verify the return values
generated by writing non-contiguous CBMs don't contradict the
reported non-contiguous support information.
Use a logical XOR to confirm return value of write_schemata() and
non-contiguous CBMs support information match.
Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 5339792bd64bd1605d3d51f8e52e94dfc385959b
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Fri Feb 16 09:34:56 2024 +0100
selftests/resctrl: Add test groups and name L3 CAT test L3_CAT
To select test to run -t parameter can be used. However, -t cat
currently maps to L3 CAT test which will be confusing after more CAT
related tests will be added.
Allow selecting tests as groups and call L3 CAT test "L3_CAT", "CAT"
group will enable all CAT related tests.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit c603ff5bb830b8c22dae56ca3ca5ceb5c103525b
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Fri Dec 15 17:05:11 2023 +0200
selftests/resctrl: Introduce generalized test framework
Each test currently has a "run test" function in per test file and
another resctrl_tests.c. The functions in resctrl_tests.c are almost
identical.
Generalize the one in resctrl_tests.c such that it can be shared
between all of the tests. It makes adding new tests easier and removes
the per test if () forests.
Also add comment to CPU vendor IDs that they must be defined as bits
for a bitmask.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 15f298821289d3efba87bb34db29d0ba9780a443
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Fri Dec 15 17:05:10 2023 +0200
selftests/resctrl: Create struct for input parameters
resctrl_tests reads a set of parameters and passes them individually
for each tests which causes variations in the call signature between
the tests.
Add struct input_params to hold all input parameters. It can be easily
passed to every test without varying the call signature.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 90a009db09e2b91990df28e47844d44ae0ef9aa8
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Fri Dec 15 17:05:06 2023 +0200
selftests/resctrl: Replace file write with volatile variable
The fill_buf code prevents compiler optimizating the entire read loop
away by writing the final value of the variable into a file. While it
achieves the goal, writing into a file requires significant amount of
work within the innermost test loop and also error handling.
A simpler approach is to take advantage of volatile. Writing through
a pointer to a volatile variable is enough to prevent compiler from
optimizing the write away, and therefore compiler cannot remove the
read loop either.
Add a volatile 'value_sink' into resctrl_tests.c and make fill_buf to
write into it. As a result, the error handling in fill_buf.c can be
simplified.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 06035f019422ba17e85c11e70d6d8bdbe9fa1afd
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Oct 2 12:48:12 2023 +0300
selftests/resctrl: Fix feature checks
The MBA and CMT tests expect support of other features to be able to
run.
When platform only supports MBA but not MBM, MBA test will fail with:
Failed to open total bw file: No such file or directory
When platform only supports CMT but not CAT, CMT test will fail with:
Failed to open bit mask file '/sys/fs/resctrl/info/L3/cbm_mask': No such file or directory
It leads to the test reporting test fail (even if no test was run at
all).
Extend feature checks to cover these two conditions to show these tests
were skipped rather than failed.
Fixes: ee0415681e ("selftests/resctrl: Use resctrl/info for feature detection")
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Cc: <stable@vger.kernel.org> # selftests/resctrl: Refactor feature check to use resource and feature name
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit d56e5da0e0f557a206bace16bbbdad00a5800e34
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Oct 2 12:48:11 2023 +0300
selftests/resctrl: Refactor feature check to use resource and feature name
Feature check in validate_resctrl_feature_request() takes in the test
name string and maps that to what to check per test.
Pass resource and feature names to validate_resctrl_feature_request()
directly rather than deriving them from the test name inside the
function which makes the feature check easier to extend for new test
cases.
Use !! in the return statement to make the boolean conversion more
obvious even if it is not strictly necessary from correctness point of
view (to avoid it looking like the function is returning a freed
pointer).
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Cc: <stable@vger.kernel.org> # selftests/resctrl: Remove duplicate feature check from CMT test
Cc: <stable@vger.kernel.org> # selftests/resctrl: Move _GNU_SOURCE define into Makefile
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 3aff5146445582454c35900f3c0c972987cdd595
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Oct 2 12:48:08 2023 +0300
selftests/resctrl: Extend signal handler coverage to unmount on receiving signal
Unmounting resctrl FS has been moved into the per test functions in
resctrl_tests.c by commit caddc0fbe495 ("selftests/resctrl: Move
resctrl FS mount/umount to higher level"). In case a signal (SIGINT,
SIGTERM, or SIGHUP) is received, the running selftest is aborted by
ctrlc_handler() which then unmounts resctrl fs before exiting. The
current section between signal_handler_register() and
signal_handler_unregister(), however, does not cover the entire
duration when resctrl FS is mounted.
Move signal_handler_register() and signal_handler_unregister() calls
from per test files into resctrl_tests.c to properly unmount resctrl
fs. In order to not add signal_handler_register()/unregister() n times,
create helpers test_prepare() and test_cleanup().
Do not call ksft_exit_fail_msg() in test_prepare() but only in the per
test function to keep the control flow cleaner without adding calls to
exit() deep into the call chain.
Adjust child process kill() call in ctrlc_handler() to only be invoked
if the child was already forked.
Fixes: caddc0fbe495 ("selftests/resctrl: Move resctrl FS mount/umount to higher level")
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit f23c7925e98a1f0d9e3ee93a856dfe3d1288a3f0
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:39 2023 +0300
selftests/resctrl: Cleanup benchmark argument parsing
Benchmark argument is handled by custom argument parsing code which is
more complicated than it needs to be.
Process benchmark argument within the normal getopt() handling and drop
unnecessary ben_ind and has_ben variables. When -b is given, terminate
the argument processing as -b consumes all remaining arguments.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 149ff729538aabaa85c44dc58a45fb99798e5e7e
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:38 2023 +0300
selftests/resctrl: Remove ben_count variable
ben_count is only used to write the terminator for the list. It is
enough to use i from the loop so no need for another variable.
Remove ben_count variable as it is not needed.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit e33cb5702a9f287d829b0e9e6abe57f6a4aba6d2
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:37 2023 +0300
selftests/resctrl: Make benchmark command const and build it with pointers
Benchmark command is used in multiple tests so it should not be
mutated by the tests but CMT test alters span argument. Due to the
order of tests (CMT test runs last), mutating the span argument in CMT
test does not trigger any real problems currently.
Mark benchmark_cmd strings as const and setup the benchmark command
using pointers. Because the benchmark command becomes const, the input
arguments can be used directly. Besides being simpler, using the input
arguments directly also removes the internal size restriction.
CMT test has to create a copy of the benchmark command before altering
the benchmark command.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 47809eb70cdc81221e03baafe8c1768801a9184e
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:36 2023 +0300
selftests/resctrl: Reorder resctrl FS prep code and benchmark_cmd init
Benchmark command is initialized before resctrl FS check and
preparation code that can call ksft_exit_skip(). There is no strong
reason why the resctrl FS support check and unmounting it (if already
mounted), has to be done after the benchmark command initialization.
Move benchmark command initialization such that it is done not until
right before the tests commence. This simplifies rollback handling when
benchmark command initialization starts to use dynamic allocation (in a
change following this).
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit b1a901e078c4ee4a6fe13021c4577ef5f3155251
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:35 2023 +0300
selftests/resctrl: Simplify span lifetime
struct resctrl_val_param contains span member. resctrl_val(), however,
never uses it because the value of span is embedded into the default
benchmark command and parsed from it by run_benchmark().
Remove span from resctrl_val_param. Provide DEFAULT_SPAN for the code
that needs it. CMT and CAT tests communicate span that is different
from the DEFAULT_SPAN between their internal functions which is
converted into passing it directly as a parameter.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 47e36f16c7846bf3627ff68525e02555c53dc99e
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:34 2023 +0300
selftests/resctrl: Remove bw_report and bm_type from main()
bw_report is always set to "reads" and bm_type is set to "fill_buf" but
is never used.
Set bw_report directly to "reads" in MBA/MBM test and remove bm_type.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 5eb6360eeeb661c7589bc344c1c51e9dfcb63451
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:33 2023 +0300
selftests/resctrl: Correct benchmark command help
Benchmark command must be the last argument because it consumes all the
remaining arguments but help misleadingly shows it as the first
argument. The benchmark command is also shown in quotes but it does not
match with the code.
Correct -b argument place in the help message and remove the quotes.
Tweak also how the options are presented by using ... notation.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 4a28c7665c2a1ac0400864eabb0c641e135f61aa
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Sep 4 12:53:32 2023 +0300
selftests/resctrl: Ensure the benchmark commands fits to its array
Benchmark command is copied into an array in the stack. The array is
BENCHMARK_ARGS items long but the command line could try to provide a
longer command. Argument size is also fixed by BENCHMARK_ARG_SIZE (63
bytes of space after fitting the terminating \0 character) and user
could have inputted argument longer than that.
Return error in case the benchmark command does not fit to the space
allocated for it.
Fixes: ecdbb911f2 ("selftests/resctrl: Add MBM test")
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: "Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 7f3c980c7738d0624077f926216bc7ac4cefcf60
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:15:03 2023 +0300
selftests/resctrl: Don't pass test name to fill_buf
Test name is passed to fill_buf functions so that they can loop around
buffer only once. This is required for CAT test case.
To loop around buffer only once, caller doesn't need to let fill_buf
know which test case it is. Instead, pass a boolean argument 'once'
which makes fill_buf more generic.
As run_benchmark() no longer needs to pass the test name to
run_fill_buf(), a few test running functions can be simplified to not
write the test name into the default benchmark_cmd. The has_ben
argument can also be removed now from those test running functions.
Co-developed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 326baed260e2e4f2df18d42d41b261740c6b8c04
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:15:00 2023 +0300
selftests/resctrl: Remove "malloc_and_init_memory" param from run_fill_buf()
run_fill_buf()'s malloc_and_init_memory parameter is always 1. There's
also duplicated memory init code for malloc_and_init_memory == 0 case
in fill_buf() which is unused.
Remove the malloc_and_init_memory parameter and the duplicated mem init
code.
While at it, fix also a typo in run_fill_buf() prototype's argument.
Co-developed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 1b9537baf8513d605a2de6c5ea218ca936418892
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:14:59 2023 +0300
selftests/resctrl: Remove duplicated preparation for span arg
When no benchmark_cmd is given, benchmark_cmd[1] is set to span in
main(). There's no need to do it again in run_mba_test().
Remove the duplicated preparation for span argument into
benchmark_cmd[1] from run_mba_test(). After this, the has_ben and span
arguments to run_mba_test() can be removed.
Co-developed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit e24ac42889810168f7a2022b868bb4b9a9481760
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:14:58 2023 +0300
selftests/resctrl: Express span internally in bytes
MBA and MBM tests to use megabytes to represent span. CMT test uses
bytes. The difference requires run_benchmark() to size the buffer
differently based on the test name, which in turn requires passing the
test name into run_benchmark().
Convert MBA and MBM tests to use span internally in bytes like CMT test
to remove the internal inconsistency between the tests. Remove the test
dependent buffer sizing from run_benchmark().
This change eliminates one of the reasons why the test name has to be
passed around but there are still other users too so the test name
passing cannot yet be removed.
Co-developed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 67a86643db2b12408430d1d627b3dc6208f06765
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:14:57 2023 +0300
selftests/resctrl: Convert span to size_t
Span is defined either as unsigned long or int.
Consistently use size_t everywhere for span as it refers to size of the
memory block.
Co-developed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 6383851a193b76dbd3ec09b91154e2dda6c96308
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:14:55 2023 +0300
selftests/resctrl: Refactor remount_resctrl(bool mum_resctrlfs) to mount_resctrl()
Mount/umount of the resctrl FS is now paired nicely per test.
Rename remount_resctrl(bool mum_resctrlfs) to mount_resctrl(). Make
it unconditionally try to mount the resctrl FS and return error if
resctrl FS was mounted already.
While at it, group the mount/umount prototypes in the header.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit caddc0fbe4958a84b0dcd257439e9090c8fbfb8e
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:14:54 2023 +0300
selftests/resctrl: Move resctrl FS mount/umount to higher level
A few places currently lack umounting resctrl FS on error paths:
- cmt_resctrl_val() has multiple error paths with direct return.
- cat_perf_miss_val() has multiple error paths with direct return.
In addition, validate_resctrl_feature_request() is called by
run_mbm_test() and run_mba_test(). Neither MBA nor MBM test tries to
umount resctrl FS.
Each and every test does require resctrl FS to be present already for
feature check. Thus, it makes sense to just mount it on higher level in
resctrl_tests.c and properly pair it with umount.
Move resctrl FS (re)mount/unmount into each test function in
resctrl_tests.c. Make feature validation to simply check that resctrl
FS is mounted.
As there's the final umount in main() before this change, the selftest
should not leave resctrl FS behind after the tests even if one of the
forementioned paths is taken (thus, no fixes tag).
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit ede13008489b36f2adcdf01e3f6435c7b24610c1
Author: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Date: Mon Jul 17 16:14:53 2023 +0300
selftests/resctrl: Unmount resctrl FS before starting the first test
Resctrl FS mount/remount/umount code is hard to track. Better approach
is to use mount/umount pair for each test but that assumes resctrl FS
is not mounted beforehand.
Change umount_resctrlfs() so that it can unmount resctrl FS from any
path, and enable further simplifications into mount/remount/umount
logic by unmounting resctrl FS at the start if a pre-existing
mountpoint is found.
Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan (Fujitsu) <tan.shaopeng@fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 91db4fd9019a08251a500157cf0d88de457e1a96
Author: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Date: Thu Apr 13 16:22:59 2023 +0900
selftests/resctrl: Remove duplicate codes that clear each test result file
Before exiting each test function(run_cmt/cat/mbm/mba_test()),
test results("ok","not ok") are printed by ksft_test_result() and then
temporary result files are cleaned by function
cmt/cat/mbm/mba_test_cleanup().
However, before running ksft_test_result(),
function cmt/cat/mbm/mba_test_cleanup()
has been run in each test function as follows:
cmt_resctrl_val()
cat_perf_miss_val()
mba_schemata_change()
mbm_bw_change()
Remove duplicate codes that clear each test result file,
while ensuring cleanup properly even when errors occur in each test.
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit c2b1790747a518f81071b44f31375d668e5b4b03
Author: Peter Newman <peternewman@google.com>
Date: Thu Mar 9 15:57:57 2023 +0100
selftests/resctrl: Use correct exit code when tests fail
Use ksft_finished() after running tests so that resctrl_tests doesn't
return exit code 0 when tests fail.
Consequently, report the MBA and MBM tests as skipped when running on
non-Intel hardware, otherwise resctrl_tests will exit with a failure
code.
Signed-off-by: Peter Newman <peternewman@google.com>
Reviewed-by: Shaopeng Tan <tan.shaopeng@fujitsu.com>
Tested-by: Shaopeng Tan <tan.shaopeng@fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 3531d930c36fb991668ba0644fb14e8e24d923af
Author: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Date: Wed Mar 23 17:12:24 2022 +0900
selftests/resctrl: Fix resctrl_tests' return code to work with selftest framework
In kselftest framework, if a sub test can not run by some reasons,
the test result should be marked as SKIP rather than FAIL.
Return KSFT_SKIP(4) instead of KSFT_FAIL(1) if resctrl_tests is not run
as root or it is run on a test environment which does not support resctrl.
- ksft_exit_fail_msg(): returns KSFT_FAIL(1)
- ksft_exit_skip(): returns KSFT_SKIP(4)
Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit d577380da04e410a31e7f944b04d838ab1c8a1c3
Author: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Date: Wed Mar 23 17:09:28 2022 +0900
selftests/resctrl: Print a message if the result of MBM&CMT tests is failed on Intel CPU
According to "Intel Resource Director Technology (Intel RDT) on
2nd Generation Intel Xeon Scalable Processors Reference Manual",
When the Intel Sub-NUMA Clustering(SNC) feature is enabled,
Intel CMT and MBM counters may not be accurate.
However, there does not seem to be an architectural way to detect
if SNC is enabled.
If the result of MBM&CMT test fails on Intel CPU,
print a message to let users know a possible cause of failure.
Acked-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
JIRA: https://issues.redhat.com/browse/RHEL-20790
commit 6220f69e72a534838cffd84dce6afd777777be03
Author: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Date: Wed Mar 23 17:09:27 2022 +0900
selftests/resctrl: Extend CPU vendor detection
Currently, the resctrl_tests only has a function to detect AMD vendor.
Since when the Intel Sub-NUMA Clustering feature is enabled,
Intel CMT and MBM counters may not be accurate,
the resctrl_tests also need a function to detect Intel vendor.
And in the future, resctrl_tests will need a function to detect different
vendors, such as Arm.
Extend the function to detect Intel vendor as well. Also,
this function can be easily extended to detect other vendors.
Signed-off-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Reviewed-by: Reinette Chatre <reinette.chatre@intel.com>
Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: David Arcari <darcari@redhat.com>
Resctrl test suite accepts command line argument "-t" to specify the
unit tests to run in the test list (e.g., -t mbm,mba,cmt,cat) as
documented in the help.
When calling strtok() to parse the option, the incorrect delimiters
argument ":\t" is used. As a result, passing "-t mbm,mba,cmt,cat" throws
an invalid option error.
Fix this by using delimiters argument "," instead of ":\t" for parsing
of unit tests list. At the same time, remove the unnecessary "spaces"
between the unit tests in help documentation to prevent confusion.
Fixes: 790bf585b0 ("selftests/resctrl: Add Cache Allocation Technology (CAT) selftest")
Fixes: 78941183d1 ("selftests/resctrl: Add Cache QoS Monitoring (CQM) selftest")
Fixes: ecdbb911f2 ("selftests/resctrl: Add MBM test")
Fixes: 034c7678dd ("selftests/resctrl: Add README for resctrl tests")
Cc: stable@vger.kernel.org
Signed-off-by: Xiaochen Shen <xiaochen.shen@intel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
umount_resctrlfs() directly attempts to unmount resctrl file system without
checking if resctrl FS is already mounted or not. It returns 0 on success
and on failure it prints an error message and returns an error status.
Calling umount_resctrlfs() when resctrl FS isn't mounted will return an
error status.
There could be situations where-in the caller might not know if resctrl
FS is already mounted or not and the caller might still want to unmount
resctrl FS if it's already mounted (For example during teardown).
To support above use cases, change umount_resctrlfs() such that it now
first checks if resctrl FS is already mounted or not and unmounts resctrl
FS only if it's already mounted.
unmount resctrl FS upon exit. For example, running only mba test on a
Broadwell (BDW) machine (MBA isn't supported on BDW CPU).
This happens because validate_resctrl_feature_request() would mount resctrl
FS to check if mba is enabled on the platform or not and finds that the H/W
doesn't support mba and hence will return false to run_mba_test(). This in
turn makes the main() function return without unmounting resctrl FS.
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
There could be two reasons why a resctrl feature might not be enabled on
the platform
1. H/W might not support the feature
2. Even if the H/W supports it, the user might have disabled the feature
through kernel command line arguments
Hence, any resctrl unit test (like cmt, cat, mbm and mba) before starting
the test will first check if the feature is enabled on the platform or not.
If the feature isn't enabled, then the test returns with an error status.
For example, if MBA isn't supported on a platform and if the user tries to
run MBA, the output will look like this
ok mounting resctrl to "/sys/fs/resctrl"
not ok MBA: schemata change
But, not supporting a feature isn't a test failure. So, instead of treating
it as an error, use the SKIP directive of the TAP protocol. With the
change, the output will look as below
ok MBA # SKIP Hardware does not support MBA or MBA is disabled
Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Resctrl test suite main() function does the following things
1. Parses command line arguments passed by user
2. Some setup checks
3. Logic that calls into each unit test
4. Print result and clean up after running each unit test
Introduce wrapper functions for steps 3 and 4 to modularize the main()
function. Adding these wrapper functions makes it easier to add any logic
to each individual test.
Please note that this is a preparatory patch for the next one and no
functional changes are intended.
Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Cache related tests (like CAT and CMT) depend on a variable called
no_of_bits to run. no_of_bits defines the number of contiguous bits
that should be set in the CBM mask and a user can pass a value for
no_of_bits using -n command line argument. If a user hasn't passed any
value, it defaults to 5 (randomly chosen value).
Hard coding no_of_bits to 5 will make the cache tests fail to run on
systems that support maximum cbm mask that is less than or equal to 5 bits.
Hence, don't hard code no_of_bits value.
If a user passes a value for "no_of_bits" using -n option, use it.
Otherwise, no_of_bits is equal to half of the maximum number of bits in
the cbm mask.
Please note that CMT test is still hard coded to 5 bits. It will change in
subsequent patches that change CMT test.
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Add a missing newline to the printed help text to improve readability.
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
CMT (Cache Monitoring Technology) [1] is a H/W feature that reports cache
occupancy of a process. resctrl selftest suite has a unit test to test CMT
for LLC but the test is named as CQM (Cache Quality Monitoring).
Furthermore, the unit test source file is named as cqm_test.c and several
functions, variables, comments, preprocessors and statements widely use
"cqm" as either suffix or prefix. This rampant misusage of CQM for CMT
might confuse someone who is newly looking at resctrl selftests because
this feature is named CMT in the Intel Software Developer's Manual.
Hence, rename all the occurrences (unit test source file name, functions,
variables, comments and preprocessors) of cqm with cmt.
[1] Please see Intel SDM, Volume 3, chapter 17 and section 18 for more
information on CMT: https://software.intel.com/content/www/us/en/develop/articles/intel-sdm.html
Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
resctrl test suite accepts command line arguments (like -b, -t, -n and -p)
as documented in the help. But passing -n and -p throws an invalid option
error. This happens because -n and -p are missing in the list of
characters that getopt() recognizes as valid arguments. Hence, they are
treated as invalid options.
Fix this by adding them to the list of characters that getopt() recognizes
as valid arguments. Please note that the main() function already has the
logic to deal with the values passed as part of these arguments and hence
no changes are needed there.
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Checking resctrl features call strcmp() to compare feature strings
(e.g. "mba", "cat" etc). The checkings are error prone and don't have
good coding style. Define the constant strings in macros and call
strncmp() to solve the potential issues.
Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
Tested-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
For now, disable MBA and MBM tests for AMD. Deciding test pass/fail
is not clear right now. We can enable when we have some clarity.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Co-developed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
RESCTRL feature is supported both on Intel and AMD now. Some features
are implemented differently. Add vendor detection mechanism. Use the vendor
check where there are differences.
Signed-off-by: Babu Moger <babu.moger@amd.com>
Co-developed-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Cache Allocation Technology (CAT) selftest allocates a portion of
last level cache and starts a benchmark to read each cache
line in this portion of cache. Measure the cache misses in perf and
the misses should be equal to the number of cache lines in this
portion of cache.
We don't use CQM to calculate cache usage because some CAT enabled
platforms don't have CQM.
Co-developed-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Signed-off-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Co-developed-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Cache QoS Monitoring (CQM) selftest starts stressful cache benchmark
with specified size of memory to access the cache. Last Level cache
occupancy reported by CQM should be close to the size of the memory.
Co-developed-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Signed-off-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Co-developed-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
MBA (Memory Bandwidth Allocation) test starts a stressful memory
bandwidth benchmark and allocates memory bandwidth from 100% down
to 10% for the benchmark process. For each allocation, compare
perf IMC counter and mbm total bytes from resctrl. The difference
between the two values should be within a threshold to pass the test.
Default benchmark is built-in fill_buf. But users can specify their
own benchmark by option "-b".
We can add memory bandwidth allocation for multiple processes in the
future.
Co-developed-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Signed-off-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Co-developed-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
MBM (Memory Bandwidth Monitoring) test is the first implemented selftest.
It starts a stressful memory bandwidth benchmark and assigns the
bandwidth pid in a resctrl monitoring group. Read and compare perf IMC
counter and MBM total bytes for the benchmark. The numbers should be
close enough to pass the test.
Default benchmark is built-in fill_buf. But users can specify their
own benchmark by option "-b".
We can add memory bandwidth monitoring for multiple processes in the
future.
Co-developed-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Signed-off-by: Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>
Co-developed-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Babu Moger <babu.moger@amd.com>
Signed-off-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
David Arcari
Xiaochen Shen
Fenghua Yu
Reinette Chatre
Babu Moger