| From 48381273f8734d28ef56a5bdf1966dd8530111bc Mon Sep 17 00:00:00 2001 |
| From: Mike Kravetz <mike.kravetz@oracle.com> |
| Date: Tue, 24 May 2022 13:50:03 -0700 |
| Subject: hugetlb: fix huge_pmd_unshare address update |
| |
| From: Mike Kravetz <mike.kravetz@oracle.com> |
| |
| commit 48381273f8734d28ef56a5bdf1966dd8530111bc upstream. |
| |
| The routine huge_pmd_unshare() is passed a pointer to an address |
| associated with an area which may be unshared. If unshare is successful |
| this address is updated to 'optimize' callers iterating over huge page |
| addresses. For the optimization to work correctly, address should be |
| updated to the last huge page in the unmapped/unshared area. However, in |
| the common case where the passed address is PUD_SIZE aligned, the address |
| is incorrectly updated to the address of the preceding huge page. That |
| wastes CPU cycles as the unmapped/unshared range is scanned twice. |
| |
| Link: https://lkml.kernel.org/r/20220524205003.126184-1-mike.kravetz@oracle.com |
| Fixes: 39dde65c9940 ("shared page table for hugetlb page") |
| Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com> |
| Acked-by: Muchun Song <songmuchun@bytedance.com> |
| Cc: <stable@vger.kernel.org> |
| Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
| Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> |
| --- |
| mm/hugetlb.c | 9 ++++++++- |
| 1 file changed, 8 insertions(+), 1 deletion(-) |
| |
| --- a/mm/hugetlb.c |
| +++ b/mm/hugetlb.c |
| @@ -4593,7 +4593,14 @@ int huge_pmd_unshare(struct mm_struct *m |
| pud_clear(pud); |
| put_page(virt_to_page(ptep)); |
| mm_dec_nr_pmds(mm); |
| - *addr = ALIGN(*addr, HPAGE_SIZE * PTRS_PER_PTE) - HPAGE_SIZE; |
| + /* |
| + * This update of passed address optimizes loops sequentially |
| + * processing addresses in increments of huge page size (PMD_SIZE |
| + * in this case). By clearing the pud, a PUD_SIZE area is unmapped. |
| + * Update address to the 'last page' in the cleared area so that |
| + * calling loop can move to first page past this area. |
| + */ |
| + *addr |= PUD_SIZE - PMD_SIZE; |
| return 1; |
| } |
| #define want_pmd_share() (1) |
