mm: Free reclaimed pages indepdent of next reclaim
Invalidate dirty/writeback page and file/swap I/O for reclaiming
are asynchronous so that when page writeback is completed,
it will be rotated back into LRU tail for freeing in next reclaim.
But it would make unnecessary CPU overhead and more aging
with higher priority of reclaim than necessary thing.
This patch makes such pages instant release when I/O complete
without LRU movement so that we could reduce reclaim events.
This patch wakes up one waiting PG_writeback and then clear
PG_reclaim bit because the page could be released during
rotating so it makes slighly race with Readahead logic but
the chance would be small and no huge side-effect even though
that happens, I belive.
Signed-off-by: Minchan Kim <minchan@kernel.org>
diff --git a/mm/filemap.c b/mm/filemap.c
index c2f30ed..6e09de6 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -752,23 +752,28 @@
*/
void end_page_writeback(struct page *page)
{
+ if (!test_clear_page_writeback(page))
+ BUG();
+
+ smp_mb__after_atomic();
+ wake_up_page(page, PG_writeback);
+
/*
* TestClearPageReclaim could be used here but it is an atomic
* operation and overkill in this particular case. Failing to
* shuffle a page marked for immediate reclaim is too mild to
* justify taking an atomic operation penalty at the end of
* ever page writeback.
+ *
+ * Clearing PG_reclaim after waking up waiter is slightly racy.
+ * Readahead might see PageReclaim as PageReadahead marker
+ * so readahead logic might be broken temporally but it isn't
+ * matter enough to care.
*/
if (PageReclaim(page)) {
ClearPageReclaim(page);
rotate_reclaimable_page(page);
}
-
- if (!test_clear_page_writeback(page))
- BUG();
-
- smp_mb__after_atomic();
- wake_up_page(page, PG_writeback);
}
EXPORT_SYMBOL(end_page_writeback);
diff --git a/mm/swap.c b/mm/swap.c
index 3074210..d61b878 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -443,6 +443,27 @@
if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) {
enum lru_list lru = page_lru_base_type(page);
+ struct address_space *mapping;
+
+ if (!trylock_page(page))
+ goto move_tail;
+
+ mapping = page_mapping(page);
+ if (!mapping)
+ goto unlock;
+
+ /*
+ * If it is successful, aotmic_remove_mapping
+ * makes page->count one so the page will be
+ * released when caller release his refcount.
+ */
+ if (atomic_remove_mapping(mapping, page)) {
+ unlock_page(page);
+ return;
+ }
+unlock:
+ unlock_page(page);
+move_tail:
list_move_tail(&page->lru, &lruvec->lists[lru]);
(*pgmoved)++;
}
