bug #5048
Updated by Andreas Kohlbecker over 7 years ago
Our cdm-server are suffering from memory problems! Evidence for this can bee seen st least since the middle of last week. * edit-int was consuming about 12GB of RAM with only 4 instances running. This became obvious because the server was swapping and became more or less unresponsive. After a restart RAM consumption was at 5GB, * edit-test was today also consuming about 12GB of RAM but with over 20 instances. It also was stalled due to swapping memory to the disk. * edit-production spit out a lot of monit warnings that the server reached the waring limit of more that 80% RAM being consumed. see also #5375 ---- Links (potentially useful): **MALLOC_ARENA Issue:** * http://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used#28935176 * http://serverfault.com/questions/341579/what-consumes-memory-in-java-process#answer-554697 * http://stackoverflow.com/questions/18734389/huge-memory-allocated-outside-of-java-heap * http://stackoverflow.com/questions/26041117/growing-resident-memory-usage-rss-of-java-process * https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en The MALLOC_ARENA Issue is a known problem with glibc >= 2.10 your servers running jessie have glibc 2.19 installed! Seting MALLOC_ARENA_MAX env variable to a low value (0-4) could help ~~~ export MALLOC_ARENA_MAX=4 ~~~ but only lead to a decrease in performance but did not reduce memory consumption at all ---- **Diagnosis of edit-test as of 30.06.2015** settings for the cdmserver: ~~~ -Xmx4500M -XX:PermSize=512m -XX:MaxPermSize=1800m ~~~ system memory usage of the cdm-server process: ~~~ KiB Mem: 10266200 total, 9720060 used, 546140 free, 95836 buffers KiB Swap: 2097148 total, 1870592 used, 226556 free. 1442588 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31761 cdm 20 0 18.997g 6.710g 3084 S 0.0 68.5 42:35.69 jsvc ~~~ **Almost all of the Swap space is being used.** ~~~ $ jmap -heap 31761 Attaching to process ID 31761, please wait... Debugger attached successfully. Server compiler detected. JVM version is 24.71-b01 using thread-local object allocation. Garbage-First (G1) GC with 4 thread(s) Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 4718592000 (4500.0MB) NewSize = 1363144 (1.2999954223632812MB) MaxNewSize = 17592186044415 MB OldSize = 5452592 (5.1999969482421875MB) NewRatio = 2 SurvivorRatio = 8 PermSize = 536870912 (512.0MB) MaxPermSize = 1887436800 (1800.0MB) G1HeapRegionSize = 1048576 (1.0MB) Heap Usage: G1 Heap: regions = 4380 capacity = 4592762880 (4380.0MB) used = 3908545928 (3727.479866027832MB) free = 684216952 (652.520133972168MB) 85.10228004629754% used G1 Young Generation: Eden Space: regions = 602 capacity = 874512384 (834.0MB) used = 631242752 (602.0MB) free = 243269632 (232.0MB) 72.18225419664269% used Survivor Space: regions = 10 capacity = 10485760 (10.0MB) used = 10485760 (10.0MB) free = 0 (0.0MB) 100.0% used G1 Old Generation: regions = 3116 capacity = 3707764736 (3536.0MB) used = 3266817416 (3115.479866027832MB) free = 440947320 (420.52013397216797MB) 88.10746227454275% used Perm Generation: capacity = 1504706560 (1435.0MB) used = 1504138704 (1434.4584503173828MB) free = 567856 (0.5415496826171875MB) 99.96226134615908% used 59361 interned Strings occupying 5910712 bytes. ~~~ so this is in total ~10GB of heap capacity for this process whereas G1 Old Generation and Perm Generation are together using ~5GB alone!!!