tree 7dd62890dfea6bf9ccb9db29b9887f0b014fb900
parent 1cb4202ed2cbd591734f6b7fb1d2b8aa8eb5a1c3
author Ihor Solodrai <ihor.solodrai@pm.me> 1734750290 +0000
committer Arnaldo Carvalho de Melo <acme@redhat.com> 1736438630 -0300

dwarves: Increase cu->obstack chunk size to 128Kb

In dwarf_loader with growing nr_jobs the wall-clock time of BTF encoding
starts worsening after a certain point [1].

While some overhead of additional threads is expected, it's not supposed
to be noticeable unless nr_jobs is set to an unreasonably big value.

It turns out when there are "too many" threads decoding DWARF, they
start competing for memory allocation: significant number of cycles is
spent in osq_lock - in the depth of malloc called within cu__zalloc.
Which suggests that many threads are trying to allocate memory at the
same time.

See an example on a perf flamegraph for run with -j240 [2]. This is
12-core machine, so the effect is small. On machines with more cores
this problem is worse.

Increasing the chunk size of obstacks associated with CUs helps to
reduce the performance penalty caused by this race condition.

[1] https://lore.kernel.org/dwarves/C82bYTvJaV4bfT15o25EsBiUvFsj5eTlm17933Hvva76CXjIcu3gvpaOCWPgeZ8g3cZ-RMa8Vp0y1o_QMR2LhPB-LEUYfZCGuCfR_HvkIP8=@pm.me/
[2] https://gist.github.com/theihor/926af22417a78605fec8d85e1338920e

Committer notes:

Alan asked and Ihor provided additional details:

  > Is this because starting with a larger obstack size means we don't have
  > to keep reallocating as the obstack grows?

  Yes. Bigger obstack size leads to lower number of malloc calls. The
  mallocs tend to happen at the same time between threads in the case of
  DWARF decoding.

  Curiously, setting a higher obstack chunk size (like 1Mb), does not
  improve the overall wall-clock time, and can even make it worse.
  This happens because the kernel takes a different code path to allocate
  bigger chunks of memory. And also most CUs are not big (at least in case
  of vmlinux), so a bigger chunk size probably increases wasted memory.

  128Kb seems to be close to a sweet spot for the vmlinux.
  The default is 4Kb

Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Ihor Solodrai <ihor.solodrai@pm.me>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Eduard Zingerman <eddyz87@gmail.com>
Cc: Mykola Lysenko <mykolal@fb.com>
Link: https://lore.kernel.org/r/20241221030445.33907-1-ihor.solodrai@pm.me
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>