direct-io: get rid of the regular bio list and associated lock

For direct-io, the dio bio lock is quite contended. It's a classic
case of submit vs completion contention, which prevents us from
scaling with O_DIRECT on bigger systems and/or faster devices.

Change the dio refcount to an atomic_t and the dio bio list to a
lockless list.

For dio_await_completion(), don't reap events individually. The only
use case we have is reaping all events, so set it up like that instead.

Use memset() for dio_submit structure clear, instead of zero filling
it on the stack. Has been shown to improve cache utilization.

Signed-off-by: Jens Axboe <>
2 files changed