[Cosmic][PATCH 0/4] Fixes for LP:1730836

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Cosmic][PATCH 0/4] Fixes for LP:1730836

Joseph Salisbury-3
BugLink: https://bugs.launchpad.net/bugs/1730836

== Cosmic Justification ==
Intel is requesting these for fixes in Cosmic.  The commits are all in mainline
as of v4.19-rc1.

Huge page helps to reduce TLB miss rate, but it has higher cache footprint,
sometimes this may cause some issue. For example, when clearing huge page
on x86_64 platform, the cache footprint is 2M. But on a Xeon E5 v3 2699
CPU, there are 18 cores, 36 threads, and only 45M LLC (last level cache).
That is, in average, there are 2.5M LLC for each core and 1.25M LLC for
each thread. If the cache pressure is heavy when clearing the huge page,
and we clear the huge page from the begin to the end, it is possible that
the begin of huge page is evicted from the cache after we finishing
clearing the end of the huge page. And it is possible for the application
to access the begin of the huge page after clearing the huge page.

To help the above situation, when we clear or copy a huge page, the order
 to clear or copy the sub-pages can be changed to clear or copy the sub-page to access last.  

== Fixes ==
c6ddfb6c5890 ("mm, clear_huge_page: move order algorithm into a separate function")
c9f4cd713835 ("mm, huge page: copy target sub-page last when copy huge page")
5b7a1d406062 ("mm, hugetlbfs: rename address to haddr in hugetlb_cow()")
974e6d66b6b5 ("mm, hugetlbfs: pass fault address to cow handler")

Huang Ying (4):
  mm, clear_huge_page: move order algorithm into a separate function
  mm, huge page: copy target sub-page last when copy huge page
  mm, hugetlbfs: rename address to haddr in hugetlb_cow()
  mm, hugetlbfs: pass fault address to cow handler

 include/linux/mm.h |   3 +-
 mm/huge_memory.c   |   3 +-
 mm/hugetlb.c       |  23 +++++-----
 mm/memory.c        | 120 +++++++++++++++++++++++++++++++++++------------------
 4 files changed, 94 insertions(+), 55 deletions(-)

--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Cosmic][PATCH 1/4] mm, clear_huge_page: move order algorithm into a separate function

Joseph Salisbury-3
From: Huang Ying <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1730836

Patch series "mm, huge page: Copy target sub-page last when copy huge
page", v2.

Huge page helps to reduce TLB miss rate, but it has higher cache
footprint, sometimes this may cause some issue.  For example, when
copying huge page on x86_64 platform, the cache footprint is 4M.  But on
a Xeon E5 v3 2699 CPU, there are 18 cores, 36 threads, and only 45M LLC
(last level cache).  That is, in average, there are 2.5M LLC for each
core and 1.25M LLC for each thread.

If the cache contention is heavy when copying the huge page, and we copy
the huge page from the begin to the end, it is possible that the begin
of huge page is evicted from the cache after we finishing copying the
end of the huge page.  And it is possible for the application to access
the begin of the huge page after copying the huge page.

In c79b57e462b5d ("mm: hugetlb: clear target sub-page last when clearing
huge page"), to keep the cache lines of the target subpage hot, the
order to clear the subpages in the huge page in clear_huge_page() is
changed to clearing the subpage which is furthest from the target
subpage firstly, and the target subpage last.  The similar order
changing helps huge page copying too.  That is implemented in this
patchset.

The patchset is a generic optimization which should benefit quite some
workloads, not for a specific use case.  To demonstrate the performance
benefit of the patchset, we have tested it with vm-scalability run on
transparent huge page.

With this patchset, the throughput increases ~16.6% in vm-scalability
anon-cow-seq test case with 36 processes on a 2 socket Xeon E5 v3 2699
system (36 cores, 72 threads).  The test case set
/sys/kernel/mm/transparent_hugepage/enabled to be always, mmap() a big
anonymous memory area and populate it, then forked 36 child processes,
each writes to the anonymous memory area from the begin to the end, so
cause copy on write.  For each child process, other child processes
could be seen as other workloads which generate heavy cache pressure.
At the same time, the IPC (instruction per cycle) increased from 0.63 to
0.78, and the time spent in user space is reduced ~7.2%.

This patch (of 4):

In c79b57e462b5d ("mm: hugetlb: clear target sub-page last when clearing
huge page"), to keep the cache lines of the target subpage hot, the
order to clear the subpages in the huge page in clear_huge_page() is
changed to clearing the subpage which is furthest from the target
subpage firstly, and the target subpage last.  This optimization could
be applied to copying huge page too with the same order algorithm.  To
avoid code duplication and reduce maintenance overhead, in this patch,
the order algorithm is moved out of clear_huge_page() into a separate
function: process_huge_page().  So that we can use it for copying huge
page too.

This will change the direct calls to clear_user_highpage() into the
indirect calls.  But with the proper inline support of the compilers,
the indirect call will be optimized to be the direct call.  Our tests
show no performance change with the patch.

This patch is a code cleanup without functionality change.

Link: http://lkml.kernel.org/r/20180524005851.4079-2-ying.huang@...
Signed-off-by: "Huang, Ying" <[hidden email]>
Suggested-by: Mike Kravetz <[hidden email]>
Reviewed-by: Mike Kravetz <[hidden email]>
Cc: Andi Kleen <[hidden email]>
Cc: Jan Kara <[hidden email]>
Cc: Michal Hocko <[hidden email]>
Cc: Andrea Arcangeli <[hidden email]>
Cc: "Kirill A. Shutemov" <[hidden email]>
Cc: Matthew Wilcox <[hidden email]>
Cc: Hugh Dickins <[hidden email]>
Cc: Minchan Kim <[hidden email]>
Cc: Shaohua Li <[hidden email]>
Cc: Christopher Lameter <[hidden email]>
Signed-off-by: Andrew Morton <[hidden email]>
Signed-off-by: Linus Torvalds <[hidden email]>
(cherry picked from commit c6ddfb6c58903262d2d77042c41dba58cf775d88)
Signed-off-by: Joseph Salisbury <[hidden email]>
---
 mm/memory.c | 90 ++++++++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 56 insertions(+), 34 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index fe497ce..3bd7ed5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4576,71 +4576,93 @@ EXPORT_SYMBOL(__might_fault);
 #endif
 
 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
-static void clear_gigantic_page(struct page *page,
- unsigned long addr,
- unsigned int pages_per_huge_page)
-{
- int i;
- struct page *p = page;
-
- might_sleep();
- for (i = 0; i < pages_per_huge_page;
-     i++, p = mem_map_next(p, page, i)) {
- cond_resched();
- clear_user_highpage(p, addr + i * PAGE_SIZE);
- }
-}
-void clear_huge_page(struct page *page,
-     unsigned long addr_hint, unsigned int pages_per_huge_page)
+/*
+ * Process all subpages of the specified huge page with the specified
+ * operation.  The target subpage will be processed last to keep its
+ * cache lines hot.
+ */
+static inline void process_huge_page(
+ unsigned long addr_hint, unsigned int pages_per_huge_page,
+ void (*process_subpage)(unsigned long addr, int idx, void *arg),
+ void *arg)
 {
  int i, n, base, l;
  unsigned long addr = addr_hint &
  ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1);
 
- if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) {
- clear_gigantic_page(page, addr, pages_per_huge_page);
- return;
- }
-
- /* Clear sub-page to access last to keep its cache lines hot */
+ /* Process target subpage last to keep its cache lines hot */
  might_sleep();
  n = (addr_hint - addr) / PAGE_SIZE;
  if (2 * n <= pages_per_huge_page) {
- /* If sub-page to access in first half of huge page */
+ /* If target subpage in first half of huge page */
  base = 0;
  l = n;
- /* Clear sub-pages at the end of huge page */
+ /* Process subpages at the end of huge page */
  for (i = pages_per_huge_page - 1; i >= 2 * n; i--) {
  cond_resched();
- clear_user_highpage(page + i, addr + i * PAGE_SIZE);
+ process_subpage(addr + i * PAGE_SIZE, i, arg);
  }
  } else {
- /* If sub-page to access in second half of huge page */
+ /* If target subpage in second half of huge page */
  base = pages_per_huge_page - 2 * (pages_per_huge_page - n);
  l = pages_per_huge_page - n;
- /* Clear sub-pages at the begin of huge page */
+ /* Process subpages at the begin of huge page */
  for (i = 0; i < base; i++) {
  cond_resched();
- clear_user_highpage(page + i, addr + i * PAGE_SIZE);
+ process_subpage(addr + i * PAGE_SIZE, i, arg);
  }
  }
  /*
- * Clear remaining sub-pages in left-right-left-right pattern
- * towards the sub-page to access
+ * Process remaining subpages in left-right-left-right pattern
+ * towards the target subpage
  */
  for (i = 0; i < l; i++) {
  int left_idx = base + i;
  int right_idx = base + 2 * l - 1 - i;
 
  cond_resched();
- clear_user_highpage(page + left_idx,
-    addr + left_idx * PAGE_SIZE);
+ process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg);
  cond_resched();
- clear_user_highpage(page + right_idx,
-    addr + right_idx * PAGE_SIZE);
+ process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg);
  }
 }
 
+static void clear_gigantic_page(struct page *page,
+ unsigned long addr,
+ unsigned int pages_per_huge_page)
+{
+ int i;
+ struct page *p = page;
+
+ might_sleep();
+ for (i = 0; i < pages_per_huge_page;
+     i++, p = mem_map_next(p, page, i)) {
+ cond_resched();
+ clear_user_highpage(p, addr + i * PAGE_SIZE);
+ }
+}
+
+static void clear_subpage(unsigned long addr, int idx, void *arg)
+{
+ struct page *page = arg;
+
+ clear_user_highpage(page + idx, addr);
+}
+
+void clear_huge_page(struct page *page,
+     unsigned long addr_hint, unsigned int pages_per_huge_page)
+{
+ unsigned long addr = addr_hint &
+ ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1);
+
+ if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) {
+ clear_gigantic_page(page, addr, pages_per_huge_page);
+ return;
+ }
+
+ process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page);
+}
+
 static void copy_user_gigantic_page(struct page *dst, struct page *src,
     unsigned long addr,
     struct vm_area_struct *vma,
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Cosmic][PATCH 2/4] mm, huge page: copy target sub-page last when copy huge page

Joseph Salisbury-3
In reply to this post by Joseph Salisbury-3
From: Huang Ying <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1730836

Huge page helps to reduce TLB miss rate, but it has higher cache
footprint, sometimes this may cause some issue.  For example, when
copying huge page on x86_64 platform, the cache footprint is 4M.  But on
a Xeon E5 v3 2699 CPU, there are 18 cores, 36 threads, and only 45M LLC
(last level cache).  That is, in average, there are 2.5M LLC for each
core and 1.25M LLC for each thread.

If the cache contention is heavy when copying the huge page, and we copy
the huge page from the begin to the end, it is possible that the begin
of huge page is evicted from the cache after we finishing copying the
end of the huge page.  And it is possible for the application to access
the begin of the huge page after copying the huge page.

In c79b57e462b5d ("mm: hugetlb: clear target sub-page last when clearing
huge page"), to keep the cache lines of the target subpage hot, the
order to clear the subpages in the huge page in clear_huge_page() is
changed to clearing the subpage which is furthest from the target
subpage firstly, and the target subpage last.  The similar order
changing helps huge page copying too.  That is implemented in this
patch.  Because we have put the order algorithm into a separate
function, the implementation is quite simple.

The patch is a generic optimization which should benefit quite some
workloads, not for a specific use case.  To demonstrate the performance
benefit of the patch, we tested it with vm-scalability run on
transparent huge page.

With this patch, the throughput increases ~16.6% in vm-scalability
anon-cow-seq test case with 36 processes on a 2 socket Xeon E5 v3 2699
system (36 cores, 72 threads).  The test case set
/sys/kernel/mm/transparent_hugepage/enabled to be always, mmap() a big
anonymous memory area and populate it, then forked 36 child processes,
each writes to the anonymous memory area from the begin to the end, so
cause copy on write.  For each child process, other child processes
could be seen as other workloads which generate heavy cache pressure.
At the same time, the IPC (instruction per cycle) increased from 0.63 to
0.78, and the time spent in user space is reduced ~7.2%.

Link: http://lkml.kernel.org/r/20180524005851.4079-3-ying.huang@...
Signed-off-by: "Huang, Ying" <[hidden email]>
Reviewed-by: Mike Kravetz <[hidden email]>
Cc: Andi Kleen <[hidden email]>
Cc: Jan Kara <[hidden email]>
Cc: Michal Hocko <[hidden email]>
Cc: Andrea Arcangeli <[hidden email]>
Cc: "Kirill A. Shutemov" <[hidden email]>
Cc: Matthew Wilcox <[hidden email]>
Cc: Hugh Dickins <[hidden email]>
Cc: Minchan Kim <[hidden email]>
Cc: Shaohua Li <[hidden email]>
Cc: Christopher Lameter <[hidden email]>
Signed-off-by: Andrew Morton <[hidden email]>
Signed-off-by: Linus Torvalds <[hidden email]>
(cherry picked from commit c9f4cd71383576a916e7fca99c490fc92a289f5a)
Signed-off-by: Joseph Salisbury <[hidden email]>
---
 include/linux/mm.h |  3 ++-
 mm/huge_memory.c   |  3 ++-
 mm/memory.c        | 30 +++++++++++++++++++++++-------
 3 files changed, 27 insertions(+), 9 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index d195d57..63ea56d 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2707,7 +2707,8 @@ extern void clear_huge_page(struct page *page,
     unsigned long addr_hint,
     unsigned int pages_per_huge_page);
 extern void copy_user_huge_page(struct page *dst, struct page *src,
- unsigned long addr, struct vm_area_struct *vma,
+ unsigned long addr_hint,
+ struct vm_area_struct *vma,
  unsigned int pages_per_huge_page);
 extern long copy_huge_page_from_user(struct page *dst_page,
  const void __user *usr_src,
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 327e126..af4ddd9 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1331,7 +1331,8 @@ int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
  if (!page)
  clear_huge_page(new_page, vmf->address, HPAGE_PMD_NR);
  else
- copy_user_huge_page(new_page, page, haddr, vma, HPAGE_PMD_NR);
+ copy_user_huge_page(new_page, page, vmf->address,
+    vma, HPAGE_PMD_NR);
  __SetPageUptodate(new_page);
 
  mmun_start = haddr;
diff --git a/mm/memory.c b/mm/memory.c
index 3bd7ed5..171bfea 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4682,11 +4682,31 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src,
  }
 }
 
+struct copy_subpage_arg {
+ struct page *dst;
+ struct page *src;
+ struct vm_area_struct *vma;
+};
+
+static void copy_subpage(unsigned long addr, int idx, void *arg)
+{
+ struct copy_subpage_arg *copy_arg = arg;
+
+ copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx,
+   addr, copy_arg->vma);
+}
+
 void copy_user_huge_page(struct page *dst, struct page *src,
- unsigned long addr, struct vm_area_struct *vma,
+ unsigned long addr_hint, struct vm_area_struct *vma,
  unsigned int pages_per_huge_page)
 {
- int i;
+ unsigned long addr = addr_hint &
+ ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1);
+ struct copy_subpage_arg arg = {
+ .dst = dst,
+ .src = src,
+ .vma = vma,
+ };
 
  if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) {
  copy_user_gigantic_page(dst, src, addr, vma,
@@ -4694,11 +4714,7 @@ void copy_user_huge_page(struct page *dst, struct page *src,
  return;
  }
 
- might_sleep();
- for (i = 0; i < pages_per_huge_page; i++) {
- cond_resched();
- copy_user_highpage(dst + i, src + i, addr + i*PAGE_SIZE, vma);
- }
+ process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg);
 }
 
 long copy_huge_page_from_user(struct page *dst_page,
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Cosmic][PATCH 3/4] mm, hugetlbfs: rename address to haddr in hugetlb_cow()

Joseph Salisbury-3
In reply to this post by Joseph Salisbury-3
From: Huang Ying <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1730836

To take better advantage of general huge page copying optimization, the
target subpage address will be passed to hugetlb_cow(), then
copy_user_huge_page().  So we will use both target subpage address and
huge page size aligned address in hugetlb_cow().  To distinguish between
them, "haddr" is used for huge page size aligned address to be
consistent with Transparent Huge Page naming convention.

Now, only huge page size aligned address is used in hugetlb_cow(), so
the "address" is renamed to "haddr" in hugetlb_cow() in this patch.
Next patch will use target subpage address in hugetlb_cow() too.

The patch is just code cleanup without any functionality changes.

Link: http://lkml.kernel.org/r/20180524005851.4079-4-ying.huang@...
Signed-off-by: "Huang, Ying" <[hidden email]>
Suggested-by: Mike Kravetz <[hidden email]>
Suggested-by: Michal Hocko <[hidden email]>
Reviewed-by: Mike Kravetz <[hidden email]>
Cc: David Rientjes <[hidden email]>
Cc: Andrea Arcangeli <[hidden email]>
Cc: "Kirill A. Shutemov" <[hidden email]>
Cc: Andi Kleen <[hidden email]>
Cc: Jan Kara <[hidden email]>
Cc: Matthew Wilcox <[hidden email]>
Cc: Hugh Dickins <[hidden email]>
Cc: Minchan Kim <[hidden email]>
Cc: Shaohua Li <[hidden email]>
Cc: Christopher Lameter <[hidden email]>
Cc: "Aneesh Kumar K.V" <[hidden email]>
Cc: Punit Agrawal <[hidden email]>
Cc: Anshuman Khandual <[hidden email]>
Signed-off-by: Andrew Morton <[hidden email]>
Signed-off-by: Linus Torvalds <[hidden email]>
(cherry picked from commit 5b7a1d406062449a4d51aea1df37a73285ced1dc)
Signed-off-by: Joseph Salisbury <[hidden email]>
---
 mm/hugetlb.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 4c1a2c0..7affa9d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3508,7 +3508,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma,
  * Keep the pte_same checks anyway to make transition from the mutex easier.
  */
 static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
-       unsigned long address, pte_t *ptep,
+       unsigned long haddr, pte_t *ptep,
        struct page *pagecache_page, spinlock_t *ptl)
 {
  pte_t pte;
@@ -3526,7 +3526,7 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
  * and just make the page writable */
  if (page_mapcount(old_page) == 1 && PageAnon(old_page)) {
  page_move_anon_rmap(old_page, vma);
- set_huge_ptep_writable(vma, address, ptep);
+ set_huge_ptep_writable(vma, haddr, ptep);
  return 0;
  }
 
@@ -3550,7 +3550,7 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
  * be acquired again before returning to the caller, as expected.
  */
  spin_unlock(ptl);
- new_page = alloc_huge_page(vma, address, outside_reserve);
+ new_page = alloc_huge_page(vma, haddr, outside_reserve);
 
  if (IS_ERR(new_page)) {
  /*
@@ -3563,11 +3563,10 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
  if (outside_reserve) {
  put_page(old_page);
  BUG_ON(huge_pte_none(pte));
- unmap_ref_private(mm, vma, old_page, address);
+ unmap_ref_private(mm, vma, old_page, haddr);
  BUG_ON(huge_pte_none(pte));
  spin_lock(ptl);
- ptep = huge_pte_offset(mm, address & huge_page_mask(h),
-       huge_page_size(h));
+ ptep = huge_pte_offset(mm, haddr, huge_page_size(h));
  if (likely(ptep &&
    pte_same(huge_ptep_get(ptep), pte)))
  goto retry_avoidcopy;
@@ -3592,12 +3591,12 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
  goto out_release_all;
  }
 
- copy_user_huge_page(new_page, old_page, address, vma,
+ copy_user_huge_page(new_page, old_page, haddr, vma,
     pages_per_huge_page(h));
  __SetPageUptodate(new_page);
  set_page_huge_active(new_page);
 
- mmun_start = address & huge_page_mask(h);
+ mmun_start = haddr;
  mmun_end = mmun_start + huge_page_size(h);
  mmu_notifier_invalidate_range_start(mm, mmun_start, mmun_end);
 
@@ -3606,25 +3605,24 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
  * before the page tables are altered
  */
  spin_lock(ptl);
- ptep = huge_pte_offset(mm, address & huge_page_mask(h),
-       huge_page_size(h));
+ ptep = huge_pte_offset(mm, haddr, huge_page_size(h));
  if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) {
  ClearPagePrivate(new_page);
 
  /* Break COW */
- huge_ptep_clear_flush(vma, address, ptep);
+ huge_ptep_clear_flush(vma, haddr, ptep);
  mmu_notifier_invalidate_range(mm, mmun_start, mmun_end);
- set_huge_pte_at(mm, address, ptep,
+ set_huge_pte_at(mm, haddr, ptep,
  make_huge_pte(vma, new_page, 1));
  page_remove_rmap(old_page, true);
- hugepage_add_new_anon_rmap(new_page, vma, address);
+ hugepage_add_new_anon_rmap(new_page, vma, haddr);
  /* Make the old page be freed below */
  new_page = old_page;
  }
  spin_unlock(ptl);
  mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
 out_release_all:
- restore_reserve_on_error(h, vma, address, new_page);
+ restore_reserve_on_error(h, vma, haddr, new_page);
  put_page(new_page);
 out_release_old:
  put_page(old_page);
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

[Cosmic][PATCH 4/4] mm, hugetlbfs: pass fault address to cow handler

Joseph Salisbury-3
In reply to this post by Joseph Salisbury-3
From: Huang Ying <[hidden email]>

BugLink: https://bugs.launchpad.net/bugs/1730836

This is to take better advantage of the general huge page copying
optimization.  Where, the target subpage will be copied last to avoid
the cache lines of target subpage to be evicted when copying other
subpages.  This works better if the address of the target subpage is
available when copying huge page.  So hugetlbfs page fault handlers are
changed to pass that information to hugetlb_cow().  This will benefit
workloads which don't access the begin of the hugetlbfs huge page after
the page fault under heavy cache contention.

Link: http://lkml.kernel.org/r/20180524005851.4079-5-ying.huang@...
Signed-off-by: "Huang, Ying" <[hidden email]>
Reviewed-by: Mike Kravetz <[hidden email]>
Cc: Michal Hocko <[hidden email]>
Cc: David Rientjes <[hidden email]>
Cc: Andrea Arcangeli <[hidden email]>
Cc: "Kirill A. Shutemov" <[hidden email]>
Cc: Andi Kleen <[hidden email]>
Cc: Jan Kara <[hidden email]>
Cc: Matthew Wilcox <[hidden email]>
Cc: Hugh Dickins <[hidden email]>
Cc: Minchan Kim <[hidden email]>
Cc: Shaohua Li <[hidden email]>
Cc: Christopher Lameter <[hidden email]>
Cc: "Aneesh Kumar K.V" <[hidden email]>
Cc: Punit Agrawal <[hidden email]>
Cc: Anshuman Khandual <[hidden email]>
Signed-off-by: Andrew Morton <[hidden email]>
Signed-off-by: Linus Torvalds <[hidden email]>
(cherry picked from commit 974e6d66b6b5c6e2d6a3ccc18b2f9a0b472be5b4)
Signed-off-by: Joseph Salisbury <[hidden email]>
---
 mm/hugetlb.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 7affa9d..231f541 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3508,7 +3508,7 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma,
  * Keep the pte_same checks anyway to make transition from the mutex easier.
  */
 static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
-       unsigned long haddr, pte_t *ptep,
+       unsigned long address, pte_t *ptep,
        struct page *pagecache_page, spinlock_t *ptl)
 {
  pte_t pte;
@@ -3517,6 +3517,7 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
  int ret = 0, outside_reserve = 0;
  unsigned long mmun_start; /* For mmu_notifiers */
  unsigned long mmun_end; /* For mmu_notifiers */
+ unsigned long haddr = address & huge_page_mask(h);
 
  pte = huge_ptep_get(ptep);
  old_page = pte_page(pte);
@@ -3591,7 +3592,7 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
  goto out_release_all;
  }
 
- copy_user_huge_page(new_page, old_page, haddr, vma,
+ copy_user_huge_page(new_page, old_page, address, vma,
     pages_per_huge_page(h));
  __SetPageUptodate(new_page);
  set_page_huge_active(new_page);
--
2.7.4


--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Reply | Threaded
Open this post in threaded view
|

APPLIED: [Cosmic][PATCH 0/4] Fixes for LP:1730836

Thadeu Lima de Souza Cascardo-3
In reply to this post by Joseph Salisbury-3
Applied to cosmic master-next branch.

Thanks.
Cascardo.

Applied-to: cosmic/master-next

--
kernel-team mailing list
[hidden email]
https://lists.ubuntu.com/mailman/listinfo/kernel-team