commit 6f2f793fba78eb4a0d5a34a71bc781118ed923d3
Author: David Gow <davidgow@google.com>
Date: Wed Feb 21 17:27:14 2024 +0800
kunit: test: Log the correct filter string in executor_test
KUnit's executor_test logs the filter string in KUNIT_ASSERT_EQ_MSG(),
but passed a random character from the filter, rather than the whole
string.
This was found by annotating KUNIT_ASSERT_EQ_MSG() to let gcc validate
the format string.
Fixes: 76066f93f1df ("kunit: add tests for filtering attributes")
Signed-off-by: David Gow <davidgow@google.com>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Justin Stitt <justinstitt@google.com>
Reviewed-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
JIRA: https://issues.redhat.com/browse/RHEL-39303
Signed-off-by: Nico Pache <npache@redhat.com>
commit dce19a3fede254fd11e0013f1c6fe3bfc37a4d73
Author: David Gow <davidgow@google.com>
Date: Wed Aug 30 08:21:15 2023 +0800
kunit: test: Make filter strings in executor_test writable
KUnit's attribute filtering feature needs the filter strings passed in
to be writable, as it modifies them in-place during parsing. This works
for the filters passed on the kernel command line, but the string
literals used in the executor tests are at least theoretically read-only
(though they work on x86_64 for some reason). s390 wasn't fooled, and
crashed when these tests were run.
Use a 'char[]' instead, (and make an explicit variable for the current
filter in parse_filter_attr_test), which will store the string in a
writable segment.
Fixes: 76066f93f1df ("kunit: add tests for filtering attributes")
Closes: https://lore.kernel.org/linux-kselftest/55950256-c00a-4d21-a2c0-cf9f0e5b8a9a@roeck-us.net/
Signed-off-by: David Gow <davidgow@google.com>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
JIRA: https://issues.redhat.com/browse/RHEL-5618
Signed-off-by: Nico Pache <npache@redhat.com>
commit c95e7c05c139b1a8a51d368bde57cf20ce931a98
Author: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
Date: Mon Aug 7 12:23:54 2023 +0200
kunit: Report the count of test suites in a module
According to KTAP specification[1], results should always start from a
header that provides a TAP protocol version, followed by a test plan with
a count of items to be executed. That pattern should be followed at each
nesting level. In the current implementation of the top-most, i.e., test
suite level, those rules apply only for test suites built into the kernel,
executed and reported on boot. Results submitted to dmesg from kunit test
modules loaded later are missing those top-level headers.
As a consequence, if a kunit test module provides more than one test suite
then, without the top level test plan, external tools that are parsing
dmesg for kunit test output are not able to tell how many test suites
should be expected and whether to continue parsing after complete output
from the first test suite is collected.
Submit the top-level headers also from the kunit test module notifier
initialization callback.
v3: Fix new name of a structure moved to kunit namespace not updated in
executor_test functions (lkp@intel.com).
v2: Use kunit_exec_run_tests() (Mauro, Rae), but prevent it from
emitting the headers when called on load of non-test modules.
[1] https://docs.kernel.org/dev-tools/ktap.html#
Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: Rae Moar <rmoar@google.com>
Reviewed-by: Rae Moar <rmoar@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
JIRA: https://issues.redhat.com/browse/RHEL-5618
Signed-off-by: Nico Pache <npache@redhat.com>
commit 76066f93f1df27657eb937b7c9c091e3a6abf4db
Author: Rae Moar <rmoar@google.com>
Date: Tue Jul 25 21:25:19 2023 +0000
kunit: add tests for filtering attributes
Add four tests to executor_test.c to test behavior of filtering attributes.
- parse_filter_attr_test - to test the parsing of inputted filters
- filter_attr_test - to test the filtering procedure on attributes
- filter_attr_empty_test - to test the behavior when all tests are filtered
out
- filter_attr_skip_test - to test the configurable filter_action=skip
option
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
JIRA: https://issues.redhat.com/browse/RHEL-5618
Signed-off-by: Nico Pache <npache@redhat.com>
commit 529534e8cba3e60f843a682e2a3149612b30d608
Author: Rae Moar <rmoar@google.com>
Date: Tue Jul 25 21:25:15 2023 +0000
kunit: Add ability to filter attributes
Add filtering of test attributes. Users can filter tests using the
module_param called "filter".
Filters are imputed in the format: <attribute_name><operation><value>
Example: kunit.filter="speed>slow"
Operations include: >, <, >=, <=, !=, and =. These operations will act the
same for attributes of the same type but may not between types.
Note multiple filters can be inputted by separating them with a comma.
Example: kunit.filter="speed=slow, module!=example"
Since both suites and test cases can have attributes, there may be
conflicts. The process of filtering follows these rules:
- Filtering always operates at a per-test level.
- If a test has an attribute set, then the test's value is filtered on.
- Otherwise, the value falls back to the suite's value.
- If neither are set, the attribute has a global "default" value, which
is used.
Filtered tests will not be run or show in output. The tests can instead be
skipped using the configurable option "kunit.filter_action=skip".
Note the default settings for running tests remains unfiltered.
Finally, add "filter" methods for the speed and module attributes to parse
and compare attribute values.
Note this filtering functionality will be added to kunit.py in the next
patch.
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Rae Moar <rmoar@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
JIRA: https://issues.redhat.com/browse/RHEL-5618
Signed-off-by: Nico Pache <npache@redhat.com>
commit 00e63f8afcfc6bf93d75141c51d35e8a40e86363
Author: David Gow <davidgow@google.com>
Date: Thu May 25 12:21:29 2023 +0800
kunit: executor_test: Use kunit_add_action()
Now we have the kunit_add_action() function, we can use it to implement
kfree_at_end() and free_subsuite_at_end() without the need for extra
helper functions.
Reviewed-by: Benjamin Berg <benjamin.berg@intel.com>
Reviewed-by: Maxime Ripard <maxime@cerno.tech>
Tested-by: Maxime Ripard <maxime@cerno.tech>
Signed-off-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
JIRA: https://issues.redhat.com/browse/RHEL-5618
Signed-off-by: Nico Pache <npache@redhat.com>
commit e5857d396f35e59e6fe96cf1178b0357cc3a1ea4
Author: Daniel Latypov <dlatypov@google.com>
Date: Sat Jul 9 11:19:58 2022 +0800
kunit: flatten kunit_suite*** to kunit_suite** in .kunit_test_suites
We currently store kunit suites in the .kunit_test_suites ELF section as
a `struct kunit_suite***` (modulo some `const`s).
For every test file, we store a struct kunit_suite** NULL-terminated array.
This adds quite a bit of complexity to the test filtering code in the
executor.
Instead, let's just make the .kunit_test_suites section contain a single
giant array of struct kunit_suite pointers, which can then be directly
manipulated. This array is not NULL-terminated, and so none of the test
filtering code needs to NULL-terminate anything.
Tested-by: Maíra Canal <maira.canal@usp.br>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Co-developed-by: David Gow <davidgow@google.com>
Signed-off-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2168378
Signed-off-by: Nico Pache <npache@redhat.com>
commit a02353f491622e49c7ddedc6a6dc4f1d6ed2150a
Author: Daniel Latypov <dlatypov@google.com>
Date: Wed May 11 14:16:26 2022 -0700
kunit: bail out of test filtering logic quicker if OOM
When filtering what tests to run (suites and/or cases) via
kunit.filter_glob (e.g. kunit.py run <glob>), we allocate copies of
suites.
These allocations can fail, and we largely don't handle that.
Note: realistically, this probably doesn't matter much.
We're not allocating much memory and this happens early in boot, so if
we can't do that, then there's likely far bigger problems.
This patch makes us immediately bail out from the top-level function
(kunit_filter_suites) with -ENOMEM if any of the underlying kmalloc()
calls return NULL.
Implementation note: we used to return NULL pointers from some functions
to indicate either that all suites/tests were filtered out or there was
an error allocating the new array.
We'll log a short error in this case and not run any tests or print a
TAP header. From a kunit.py user's perspective, they'll get a message
about missing/invalid TAP output and have to dig into the test.log to
see it. Since hitting this error seems so unlikely, it's probably fine
to not invent a way to plumb this error message more visibly.
See also: https://lore.kernel.org/linux-kselftest/20220329103919.2376818-1-lv.ruyi@zte.com.cn/
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reported-by: Zeal Robot <zealci@zte.com.cn>
Reported-by: Lv Ruyi <lv.ruyi@zte.com.cn>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2168378
Signed-off-by: Nico Pache <npache@redhat.com>
commit cd94fbc2cafb61cc1394cdba3f92b99ce07b03ae
Author: Daniel Latypov <dlatypov@google.com>
Date: Fri Oct 1 18:36:35 2021 -0700
kunit: fix too small allocation when using suite-only kunit.filter_glob
When a user filters by a suite and not a test, e.g.
$ ./tools/testing/kunit/kunit.py run 'suite_name'
it hits this code
const int len = strlen(filter_glob);
...
parsed->suite_glob = kmalloc(len, GFP_KERNEL);
which fails to allocate space for the terminating NULL.
Somehow, it seems like we can't easily reproduce this under UML, so the
existing `parse_filter_test()` didn't catch this.
Fix this by allocating `len + 1` and switch to kzalloc() just to be a
bit more defensive. We're only going to run this code once per kernel
boot, and it should never be very long.
Also update the unit tests to be a bit more cautious.
This bug showed up as a NULL pointer dereference here:
> KUNIT_EXPECT_STREQ(test, (const char *)filtered.start[0][0]->name, "suite0");
`filtered.start[0][0]` was NULL, and `name` is at offset 0 in the struct,
so `...->name` was also NULL.
Fixes: 3b29021ddd10 ("kunit: tool: allow filtering test cases via glob")
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Acked-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2168378
Signed-off-by: Nico Pache <npache@redhat.com>
commit a127b154a8f231709754b5d56a501163dd837459
Author: Daniel Latypov <dlatypov@google.com>
Date: Tue Sep 14 14:03:48 2021 -0700
kunit: tool: allow filtering test cases via glob
Commit 1d71307a6f ("kunit: add unit test for filtering suites by
names") introduced the ability to filter which suites we run via glob.
This change extends it so we can also filter individual test cases
inside of suites as well.
This is quite useful when, e.g.
* trying to run just the tests cases you've just added or are working on
* trying to debug issues with test hermeticity
Examples:
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit '*exec*.parse*'
...
============================================================
======== [PASSED] kunit_executor_test ========
[PASSED] parse_filter_test
============================================================
Testing complete. 1 tests run. 0 failed. 0 crashed.
$ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit '*.no_matching_tests'
...
[ERROR] no tests run!
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2168378
Signed-off-by: Nico Pache <npache@redhat.com>
commit f62314b1ced25c58b86e044fc951cd6a1ea234cf
Author: Xiyu Yang <xiyuyang19@fudan.edu.cn>
Date: Thu Sep 9 15:24:36 2021 +0800
kunit: fix reference count leak in kfree_at_end
The reference counting issue happens in the normal path of
kfree_at_end(). When kunit_alloc_and_get_resource() is invoked, the
function forgets to handle the returned resource object, whose refcount
increased inside, causing a refcount leak.
Fix this issue by calling kunit_alloc_resource() instead of
kunit_alloc_and_get_resource().
Fixed the following when applying:
Shuah Khan <skhan@linuxfoundation.org>
CHECK: Alignment should match open parenthesis
+ kunit_alloc_resource(test, NULL, kfree_res_free, GFP_KERNEL,
(void *)to_free);
Signed-off-by: Xiyu Yang <xiyuyang19@fudan.edu.cn>
Signed-off-by: Xin Tan <tanxin.ctf@gmail.com>
Reviewed-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2168378
Signed-off-by: Nico Pache <npache@redhat.com>
This adds unit tests for kunit_filter_subsuite() and
kunit_filter_suites().
Note: what the executor means by "subsuite" is the array of suites
corresponding to each test file.
This patch lightly refactors executor.c to avoid the use of global
variables to make it testable.
It also includes a clever `kfree_at_end()` helper that makes this test
easier to write than it otherwise would have been.
Tested by running just the new tests using itself
$ ./tools/testing/kunit/kunit.py run '*exec*'
Signed-off-by: Daniel Latypov <dlatypov@google.com>
Reviewed-by: David Gow <davidgow@google.com>
Acked-by: Brendan Higgins <brendanhiggins@google.com>
Tested-by: Brendan Higgins <brendanhiggins@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
Nico Pache
Daniel Latypov