Jiajun Yao

Stay hungry, Stay foolish.

Interaction Between HugeTLBFS and Hugepages

This post shows the interaction between hugetlbfs and huge pages by an example program. All the results are based on linux 3.10.0-514.55.4.el7.x86_64.

Setup

1
2

sudo mount -t hugetlbfs -o mode=0777,pagesize=2M nodev /mnt/huge
echo '10' | sudo tee /proc/sys/vm/nr_hugepages

This creates a hugetlbfs with maximum of 10 huge pages.

Example Program

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>

#define MB (1024*1024)

void meminfo(char* msg) {
  printf(msg);
  system("cat /proc/meminfo | grep HugePages_");
  printf("\n");
}

int main(int argc, char* argv[]) {
  meminfo("Initial meminfo: \n");

  int fd = open("/mnt/huge/foo.txt", O_RDWR | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
  meminfo("After create and open the file: \n");

  ftruncate(fd, 4 * MB);
  meminfo("After set file size to 4MB: \n");

  void* base = mmap(NULL, 4 * MB, PROT_WRITE | PROT_READ, MAP_SHARED, fd, 0);
  meminfo("After mmap 4MB without MAP_POPULATE: \n");

  munmap(base, 4 * MB);
  meminfo("After munmap: \n");

  base = mmap(NULL, 4 * MB, PROT_WRITE | PROT_READ, MAP_SHARED | MAP_POPULATE, fd, 0);
  meminfo("After mmap 4MB with MAP_POPULATE: \n");

  munmap(base, 4 * MB);
  meminfo("After munmap: \n");

  ftruncate(fd, 8 * MB);
  meminfo("After set file size to 8MB: \n");

  base = mmap(NULL, 8 * MB, PROT_WRITE | PROT_READ, MAP_SHARED | MAP_POPULATE, fd, 0);
  meminfo("After mmap 8MB with MAP_POPULATE: \n");

  munmap(base, 8 * MB);
  meminfo("After munmap: \n");

  close(fd);
  meminfo("After close: \n");
}

The output of running this program is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

Initial meminfo:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0

After creat and open the file:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0

After set file size to 4MB:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0

After mmap 4MB without MAP_POPULATE:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        2
HugePages_Surp:        0

After munmap:
HugePages_Total:      10
HugePages_Free:       10
HugePages_Rsvd:        2
HugePages_Surp:        0

After mmap 4MB with MAP_POPULATE:
HugePages_Total:      10
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0

After munmap:
HugePages_Total:      10
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0

After set file size to 8MB:
HugePages_Total:      10
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0

After mmap 8MB with MAP_POPULATE:
HugePages_Total:      10
HugePages_Free:        6
HugePages_Rsvd:        0
HugePages_Surp:        0

After munmap:
HugePages_Total:      10
HugePages_Free:        6
HugePages_Rsvd:        0
HugePages_Surp:        0

After close:
HugePages_Total:      10
HugePages_Free:        6
HugePages_Rsvd:        0
HugePages_Surp:        0

Observations

  1. Just setting the file size won’t allocate/reserve any huge pages. It only affects the logical file size not the physical one.
  2. mmap only reserves the huge pages.
  3. Populating the page table or accessing a page actually allocates the huge pages.
  4. HugePages_Free includes HugePages_Rsvd so the number of free-to-use huge pages is actually HugePages_Free minus HugePages_Rsvd.

Reference

  1. Linux下hugetlbpage使用详解

Comments