Jiajun Yao

Stay hungry, Stay foolish.

Interaction Between HugeTLBFS and Hugepages

This post shows the interaction between hugetlbfs and huge pages by a example program. All the results are based on linux 3.10.0-514.55.4.el7.x86_64.

Setup

1
2
sudo mount -t hugetlbfs -o mode=0777,pagesize=2M nodev /mnt/huge
echo '10' | sudo tee /proc/sys/vm/nr_hugepages

This creates a hugetlbfs with maximum of 10 huge pages.

Example Program

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>

#define MB (1024*1024)

void meminfo(char* msg) {
  printf(msg);
  system("cat /proc/meminfo | grep HugePages_");
  printf("\n");
}

int main(int argc, char* argv[]) {
  meminfo("Initial meminfo: \n");

  int fd = open("/mnt/huge/foo.txt", O_RDWR | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
  meminfo("After create and open the file: \n");

  ftruncate(fd, 4 * MB);
  meminfo("After set file size to 4MB: \n");

  void* base = mmap(NULL, 4 * MB, PROT_WRITE | PROT_READ, MAP_SHARED, fd, 0);
  meminfo("After mmap 4MB without MAP_POPULATE: \n");

  munmap(base, 4 * MB);
  meminfo("After munmap: \n");

  base = mmap(NULL, 4 * MB, PROT_WRITE | PROT_READ, MAP_SHARED | MAP_POPULATE, fd, 0);
  meminfo("After mmap 4MB with MAP_POPULATE: \n");

  munmap(base, 4 * MB);
  meminfo("After munmap: \n");

  ftruncate(fd, 8 * MB);
  meminfo("After set file size to 8MB: \n");

  base = mmap(NULL, 8 * MB, PROT_WRITE | PROT_READ, MAP_SHARED | MAP_POPULATE, fd, 0);
  meminfo("After mmap 8MB with MAP_POPULATE: \n");

  munmap(base, 8 * MB);
  meminfo("After munmap: \n");

  close(fd);
  meminfo("After close: \n");
}

The output of running this program is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
Initial meminfo:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0

After creat and open the file:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0

After set file size to 4MB:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0

After mmap 4MB without MAP_POPULATE:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        2
HugePages_Surp:        0

After munmap:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        2
HugePages_Surp:        0

After mmap 4MB with MAP_POPULATE:
HugePages_Total:      10
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0

After munmap:
HugePages_Total:      10
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0

After set file size to 8MB:
HugePages_Total:      10
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0

After mmap 8MB with MAP_POPULATE:
HugePages_Total:      10
HugePages_Free:        6
HugePages_Rsvd:        0
HugePages_Surp:        0

After munmap:
HugePages_Total:      10
HugePages_Free:        6
HugePages_Rsvd:        0
HugePages_Surp:        0

After close:
HugePages_Total:      10
HugePages_Free:        6
HugePages_Rsvd:        0
HugePages_Surp:        0

Observations

  1. Just setting the file size won’t allocate/reserve any huge pages. It only affects the logical file size not the physical one.
  2. mmap only reserves the huge pages.
  3. Populating the page table or accessing a page actually allocates the huge pages.
  4. HugePages_Free includes HugePages_Rsvd so the number of free-to-use huge pages is actually HugePages_Free minus HugePages_Rsvd.

Reference

  1. Linux下hugetlbpage使用详解

Comments