ORA-27102 out of memory on a system with huge memory

 

ORA-27102 out of memory on a system with huge memory

Today one customer said the data sync of Oracle Dataguard had issue and our guys checked the alert log then sent below errors to me:

ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory

Also had some ORA-00313 errors about failure to open data/redo files.

While he also confirmed the instance was opened and was running.

I asked to check the memory usage, detail trace log and system message log file continually.

The system has 200G physical memory and 16G swap while at that time we only got 3M free swap! so we did get memory issue.

Our guys checked the processes using most of the swap and killed some of them while did not fix it as only very small part of swap was freed.

I got more detailed memory info through /proc/meminfo:

[root@erpdg ~]# cat /proc/meminfo
MemTotal:       206352512 kB
MemFree:         1260976 kB
MemAvailable:     520844 kB
Buffers:            3160 kB
Cached:         41799796 kB
SwapCached:       106432 kB
Active:         40449436 kB
Inactive:        3372480 kB
......
SwapTotal:      16777212 kB
SwapFree:         108580 kB
.....
HugePages_Total:   72000
HugePages_Free:    71984
HugePages_Rsvd:      113
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      323520 kB
DirectMap2M:    24842240 kB
DirectMap1G:    186646528 kB

If you read the above output carefully, you would already know the reason.

Then I asked our guys to do below actions:

1. Check the SGA and Memory target settings
2. Stop the database instance
3. sync; echo 3 > /proc/sys/vm/drop_caches 
4. start the database instance again
5. cat /proc/meminfo again

The SGA was about 120G, and the new content of /proc/meminfo was as below:

[root@erpdg ~]# cat /proc/meminfo
MemTotal:       206352512 kB
MemFree:        44060916 kB
MemAvailable:   43266384 kB
Buffers:            4192 kB
Cached:          1122640 kB
SwapCached:        39268 kB
......
SwapTotal:      16777212 kB
SwapFree:       15826640 kB
......
HugePages_Total:   72000
HugePages_Free:    10672
HugePages_Rsvd:      113
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      323520 kB
DirectMap2M:    24842240 kB
DirectMap1G:    186646528 kB

So this issue was caused by the HugePage memory.

For some reasons, last time when the database instances was started, it did not make use of HugePage memory as there were about 140G free HugePage memory. After the above actions now we had about 20G HugePage memory. It was correct as the SGA size was about 120G.