mm: COW: skip the page lock in the COW copy path

The only guarantee required to be allowed to copy is that mapcount is
not equal 1. That is enough not to risk overestimating mapcount.

This should retain the performance improvement reported below for the
COW copy path, but without introducing any regression to the page
reuse accuracy.

https://lkml.kernel.org/r/20200914024321.GG26874@shao2-debian

In addition this optimizes the THP COW with the same logic.

lkp@lists.01.org, lkp@intel.com, https://github.com/intel/lkp-tests.git reports:

---------
FYI, we noticed a 55.2% improvement of vm-scalability.throughput due to commit:

commit: 599aa62474f51a470408b28fd4365320a5357aca ("mm: COW: skip the page lock in the COW copy path")
https://git.kernel.org/cgit/linux/kernel/git/andrea/aa.git master

in testcase: vm-scalability
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

        runtime: 300s
        size: 8T
        test: anon-cow-seq
        cpufreq_governor: performance
        ucode: 0x5003006

                              vm-scalability.throughput

  6e+07 +-------------------------------------------------------------------+
        | OO O    O      O O    O   O    O    O     O    O  O               |
  5e+07 |-+      O      O         O   OO   O O    O        O                |
        |      +. +.         O          .+.   +.O    +.   .+  O             |
        |.++   : +  +.+.++. .+.++.+.+.++   +.+  +.+.+  +.+  +.+.+.+         |
  4e+07 |-+:   :           +                                       :        |
        |  :  :                                                    +.+.+.++.|
  3e+07 |-+ : :                                                             |
        |   : :                                                             |
  2e+07 |-+ : :                                                             |
        |   : :                                                             |
        |   : :                                                             |
  1e+07 |-+  :                                                              |
        |    :                                                              |
      0 +-------------------------------------------------------------------+
---------

Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>

mm/huge_memory.c

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 656b511..7fbdf7a 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1341,6 +1341,9 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)
 	page = pmd_page(orig_pmd);
 	VM_BUG_ON_PAGE(!PageHead(page), page);
 
+	if (page_trans_huge_anon_shared(page))
+		goto copy;
+
 	/* Lock page for reuse_swap_page() */
 	if (!trylock_page(page)) {
 		get_page(page);
@@ -1372,6 +1375,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)
 	}
 
 	unlock_page(page);
+copy:
 	spin_unlock(vmf->ptl);
 fallback:
 	__split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL);

mm/memory.c

diff --git a/mm/memory.c b/mm/memory.c
index d36240c..1bbbad6 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3423,6 +3423,30 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
 	if (PageAnon(vmf->page)) {
 		struct page *page = vmf->page;
 
+		/*
+		 * Optimize away the trylock_page for mapcount > 1.
+		 *
+		 * We need to provide full accuracy and avoid spurious
+		 * COWs to avoid breaking the long term GUP pins if
+		 * the anon page is exclusive as in mapcount == 1. If
+		 * we find the mapcount at any time elevated above 1
+		 * for a non THP page, it means any GUP pin already
+		 * might have lost coherency.
+		 *
+		 * It is possible that if mapcount is found > 1 while
+		 * munmap or exit or MADV_DONTNEED in the parent is
+		 * running concurrently to the COW fault and that the
+		 * mapcount is concurrently on its way to return equal
+		 * 1, but no guarantee was provided anyway in such
+		 * case. The coherency between the GUP pin and the CPU
+		 * could have been lost if only the timing was any
+		 * different. So all it matters to avoid breaking long
+		 * term GUP pins, is that there are no false positive
+		 * COWs when mapcount is found equal 1.
+		 */
+		if (page_mapcount(vmf->page) > 1)
+			goto copy;
+
 		/* PageKsm() doesn't necessarily raise the page refcount */
 		if (PageKsm(page) || page_count(page) != 1)
 			goto copy;