kunit: tool: Add --parallel to run suites in concurrent kernels

kunit.py boots a single kernel to run all the selected tests, one after
another on a single CPU. This is good for robustness but means that the
overall time taken to complete testing could be improved. Since we are
already able to split up the set of tests run into multiple kernel boots
with the --run-isolated option let's take advantage of that and allow
execution in parallel.

Add a --parallel N option to the run and exec subcommands. When N is
greater than one the tests are enumerated with kunit.action=list and
split up by top level suite, as --run_isolated=suite does, or by
individual test if --run_isolated=test is also given. The resulting
kernels are run from a thread pool N wide, each logging its console to
test.<suite/test>.log in the build directory.

Each kernel's output is buffered by its worker and handed to the parser
in the order the suites were enumerated in, which is the order the
kernel would have run them, so the results are reported identically
regardless of which kernel happens to finish first. While developing
this I noticed some issues with the existing --run_isolated support due
to test naming, with those fixes in place results were identical with
and without parallelism.

Performance results with --alltests, using the elapsed time reported by
the KUnit runner with an already built kernel on a Raspberry Pi 5 with 4
CPUs and a Graviton 4 with 192 CPUs:

		     default  parallel  per test   parallel per test
  Raspberry Pi 5     67.7s    49.4s     836.4s     326.5s
  Graviton 4         64.1s    30.8s     445.9s      39.7s

Even with the overhead of having to boot more kernels we are seeing
substantial speedups, for obvious reasons these are much better with
more CPUs available.

Signed-off-by: Mark Brown <broonie@kernel.org>
3 files changed