| From foo@baz Mon Sep 17 12:33:31 CEST 2018 |
| From: Reza Arbab <arbab@linux.ibm.com> |
| Date: Thu, 2 Aug 2018 23:03:36 -0500 |
| Subject: powerpc/powernv: Fix concurrency issue with npu->mmio_atsd_usage |
| |
| From: Reza Arbab <arbab@linux.ibm.com> |
| |
| [ Upstream commit 9eab9901b015f489199105c470de1ffc337cfabb ] |
| |
| We've encountered a performance issue when multiple processors stress |
| {get,put}_mmio_atsd_reg(). These functions contend for |
| mmio_atsd_usage, an unsigned long used as a bitmask. |
| |
| The accesses to mmio_atsd_usage are done using test_and_set_bit_lock() |
| and clear_bit_unlock(). As implemented, both of these will require |
| a (successful) stwcx to that same cache line. |
| |
| What we end up with is thread A, attempting to unlock, being slowed by |
| other threads repeatedly attempting to lock. A's stwcx instructions |
| fail and retry because the memory reservation is lost every time a |
| different thread beats it to the punch. |
| |
| There may be a long-term way to fix this at a larger scale, but for |
| now resolve the immediate problem by gating our call to |
| test_and_set_bit_lock() with one to test_bit(), which is obviously |
| implemented without using a store. |
| |
| Fixes: 1ab66d1fbada ("powerpc/powernv: Introduce address translation services for Nvlink2") |
| Signed-off-by: Reza Arbab <arbab@linux.ibm.com> |
| Acked-by: Alistair Popple <alistair@popple.id.au> |
| Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> |
| Signed-off-by: Sasha Levin <alexander.levin@microsoft.com> |
| Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> |
| --- |
| arch/powerpc/platforms/powernv/npu-dma.c | 5 +++-- |
| 1 file changed, 3 insertions(+), 2 deletions(-) |
| |
| --- a/arch/powerpc/platforms/powernv/npu-dma.c |
| +++ b/arch/powerpc/platforms/powernv/npu-dma.c |
| @@ -427,8 +427,9 @@ static int get_mmio_atsd_reg(struct npu |
| int i; |
| |
| for (i = 0; i < npu->mmio_atsd_count; i++) { |
| - if (!test_and_set_bit_lock(i, &npu->mmio_atsd_usage)) |
| - return i; |
| + if (!test_bit(i, &npu->mmio_atsd_usage)) |
| + if (!test_and_set_bit_lock(i, &npu->mmio_atsd_usage)) |
| + return i; |
| } |
| |
| return -ENOSPC; |