| From 40c92da02ca7cbbffeb173593a59e10b3c639301 Mon Sep 17 00:00:00 2001 |
| From: Dongsheng Yang <dongsheng.yang@easystack.cn> |
| Date: Wed, 27 Jan 2021 11:15:50 +0800 |
| Subject: [PATCH] bcache: dont reset bio opf in bch_data_insert_start |
| |
| commit ad0d9e76(bcache: use bio op accessors) makes the bi_opf |
| modified by bio_set_op_attrs(). But there is a logical |
| problem in this commit: |
| |
| trace_bcache_cache_insert(k); |
| bch_keylist_push(&op->insert_keys); |
| |
| - n->bi_rw |= REQ_WRITE; |
| + bio_set_op_attrs(n, REQ_OP_WRITE, 0); |
| bch_submit_bbio(n, op->c, k, 0); |
| } while (n != bio); |
| |
| The old code add REQ_WRITE into bio n and keep other flags; the |
| new code set REQ_OP_WRITE to bi_opf, but reset all other flags. |
| |
| This problem is discoverd in our performance testing: |
| (1) start a fio with 1M x 128depth for read in /dev/nvme0n1p1 |
| (2) start a fio with 1M x 128depth for write in /dev/escache0 (cache |
| device is /dev/nvme0n1p2) |
| |
| We found the BW of reading is 2000+M/s, but the BW of writing is |
| 0-100M/s. After some debugging, we found the problem is io submit in |
| writting is very slow. |
| |
| bch_data_insert_start() insert a bio to /dev/nvme0n1p1, but as |
| cached_dev submit stack bio will be added into current->bio_list, and |
| return.Then __submit_bio_noacct() will submit the new bio in bio_list |
| into /dev/nvme0n1p1. This operation would be slow in |
| blk_mq_submit_bio() -> rq_qos_throttle(q, bio); |
| |
| The rq_qos_throttle() will call wbt_should_throttle(), |
| static inline bool wbt_should_throttle(struct rq_wb *rwb, struct bio *bio) |
| { |
| switch (bio_op(bio)) { |
| case REQ_OP_WRITE: |
| /* |
| * Don't throttle WRITE_ODIRECT |
| */ |
| if ((bio->bi_opf & (REQ_SYNC | REQ_IDLE)) == |
| (REQ_SYNC | REQ_IDLE)) |
| return false; |
| ... ... |
| } |
| |
| As the bio_set_op_attrs() reset the (REQ_SYNC | REQ_IDLE), so this write |
| bio will be considered as non-direct write. |
| |
| After this fix, bio to nvme will flaged as (REQ_SYNC | REQ_IDLE), |
| then fio for writing will get about 1000M/s bandwidth. |
| |
| Fixes: ad0d9e76a412 ("bcache: use bio op accessors") |
| Cc: Mike Christie <mchristi@redhat.com> |
| Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn> |
| Cc: stable@vger.kernel.org # 4.8+ |
| Signed-off-by: Coly Li <colyli@suse.de> |
| --- |
| drivers/md/bcache/request.c | 2 +- |
| 1 file changed, 1 insertion(+), 1 deletion(-) |
| |
| diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c |
| index 85b1f2a9b72d..57c4486f9c82 100644 |
| --- a/drivers/md/bcache/request.c |
| +++ b/drivers/md/bcache/request.c |
| @@ -244,7 +244,7 @@ static void bch_data_insert_start(struct closure *cl) |
| trace_bcache_cache_insert(k); |
| bch_keylist_push(&op->insert_keys); |
| |
| - bio_set_op_attrs(n, REQ_OP_WRITE, 0); |
| + n->bi_opf |= REQ_OP_WRITE; |
| bch_submit_bbio(n, op->c, k, 0); |
| } while (n != bio); |
| |
| -- |
| 2.26.2 |
| |