x86/retpoline: Avoid uncontrolled return buffer underflows on pre-SMEP

This patch further hardens the retpoline kernels on older systems
without SMEP.

CPUs have return buffers which store the return address for
RET to predict function returns. When the return buffer underflow
the CPU falls back to indirect branch prediction.

With retpoline we want to avoid uncontrolled indirect branches,
which could be poisoned by ring 3, so we need to avoid uncontrolled
return buffer underflows in the kernel.

CPUs have between 16 and 32 entry deep return buffers.

This is not a problem on newer CPUs (Broadwall+,Goldmont+) implementing SMEP,
which guards against this, but can happen on older CPUs or if SMEP
is disabled. On those with SMEP enabled or with RETPONLINE disabled
all the new code can be skipped.

This patch handles two scenarios where underflows could happen:

(1) The kernel and the interrupt together have a deep enough
call chain that it overflows the return buffer, which leads
to a later kernel return underflowing and falling back to the indirect
branch predictor.

To avoid this we always execute 32 unpaired rets on interrupt/NMI/
exception exit to fill the return buffer with controlled content. This
costs about 160 cycles (on Skylake) plus some mispredicts later.

This has to be done in interrupt, NMI, machine check handlers
and anything else that interrupts the kernel. It doesn't
need to be done when returning to user space, so it only
impacts kernel interruptions.

With 32 entries this scenario is not very likely, but with 16 entries
it is quite realistic.

(2) We're context switching from a shallower to a deeper kernel stack.
The deeper kernel stack would eventually underflow the return buffer,
which again would fall back to the indirect branch predictor.

To guard against this again fill the return buffer with controlled
content during context switch.

We always fill the buffer with 32 entries. That's pessimistic
because we already did some controlled kernel calls. So less dummy calls
would be needed to be safe. However it's hard to maintain such an
invariant, and it may be broken with more aggressive compilers.
So err on the side of safety and always fill completely.

Signed-off-by: Andi Kleen <ak@linux.intel.com>