bug #7938
Updated by Andreas Kohlbecker over 5 years ago
This performance problem was observed at the edit production server but might also affect other installations. Users where complaining about the server being very slow since a couple of days. The reason for the server being "slow" is that two sub processes are consuming 100% CPU time of one cpu core each. This time excessive garbage collection could be excluded as cause. ![](Screenshot_20181205_112519.png) ![](Screenshot_20181205_112425.png) The cpu load is produced by the to subprocesses with the PIDs **9527** and **9528** (main process id is 9505) A stake trace with native frames (`/opt/java-oracle/jdk1.8.0_91/bin/jstack -m -F 9505`) included give first insight into what these processes are doing (attachment:jstack-with-native-frames.tdump): ~~~ ----------------- 9527 ----------------- 0x00007fe4a30c8f10 _ZNK7MemNode20can_see_stored_valueEP4NodeP14PhaseTransform + 0x2c0 0x00007fe4a30c9c74 _ZNK8LoadNode5ValueEP14PhaseTransform + 0xf4 0x00007fe4a31ac87a _ZN12PhaseIterGVN13transform_oldEP4Node + 0xea 0x00007fe4a31a9fcc _ZN12PhaseIterGVN8optimizeEv + 0x1bc 0x00007fe4a2ce51a9 _ZN7Compile8OptimizeEv + 0x159 0x00007fe4a2ce6fea _ZN7CompileC1EP5ciEnvP10C2CompilerP8ciMethodibbb + 0x13ca 0x00007fe4a2c34818 _ZN10C2Compiler14compile_methodEP5ciEnvP8ciMethodi + 0x198 0x00007fe4a2cf14da _ZN13CompileBroker25invoke_compiler_on_methodEP11CompileTask + 0xc9a 0x00007fe4a2cf2486 _ZN13CompileBroker20compiler_thread_loopEv + 0x5d6 0x00007fe4a32ae77f _ZN10JavaThread17thread_main_innerEv + 0xdf 0x00007fe4a32ae8ac _ZN10JavaThread3runEv + 0x11c 0x00007fe4a3161f78 _ZL10java_startP6Thread + 0x108 ----------------- 9528 ----------------- 0x00007fe4a30c8ed6 _ZNK7MemNode20can_see_stored_valueEP4NodeP14PhaseTransform + 0x286 0x00007fe4a30c9c74 _ZNK8LoadNode5ValueEP14PhaseTransform + 0xf4 0x00007fe4a31ac87a _ZN12PhaseIterGVN13transform_oldEP4Node + 0xea 0x00007fe4a31a9fcc _ZN12PhaseIterGVN8optimizeEv + 0x1bc 0x00007fe4a2ce51a9 _ZN7Compile8OptimizeEv + 0x159 0x00007fe4a2ce6fea _ZN7CompileC1EP5ciEnvP10C2CompilerP8ciMethodibbb + 0x13ca 0x00007fe4a2c34818 _ZN10C2Compiler14compile_methodEP5ciEnvP8ciMethodi + 0x198 0x00007fe4a2cf14da _ZN13CompileBroker25invoke_compiler_on_methodEP11CompileTask + 0xc9a 0x00007fe4a2cf2486 _ZN13CompileBroker20compiler_thread_loopEv + 0x5d6 0x00007fe4a32ae77f _ZN10JavaThread17thread_main_innerEv + 0xdf 0x00007fe4a32ae8ac _ZN10JavaThread3runEv + 0x11c 0x00007fe4a3161f78 _ZL10java_startP6Thread + 0x108 ~~~ about 30 minutes later (attachment:jstack-verbose-with-native-frames-2.tdump) these native threads are still at the same point in the stack and seem to be stock in the ['MemNode::can_see_stored_value'](https://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/3aa636f2a743/src/share/vm/opto/memnode.cpp#l965)` c++ function: ~~~ ----------------- 9527 ----------------- 0x00007fe4a30c8f10 _ZNK7MemNode20can_see_stored_valueEP4NodeP14PhaseTransform + 0x2c0 0x00007fe4a30c9c74 _ZNK8LoadNode5ValueEP14PhaseTransform + 0xf4 0x00007fe4a31ac87a _ZN12PhaseIterGVN13transform_oldEP4Node + 0xea 0x00007fe4a31a9fcc _ZN12PhaseIterGVN8optimizeEv + 0x1bc 0x00007fe4a2ce51a9 _ZN7Compile8OptimizeEv + 0x159 0x00007fe4a2ce6fea _ZN7CompileC1EP5ciEnvP10C2CompilerP8ciMethodibbb + 0x13ca 0x00007fe4a2c34818 _ZN10C2Compiler14compile_methodEP5ciEnvP8ciMethodi + 0x198 0x00007fe4a2cf14da _ZN13CompileBroker25invoke_compiler_on_methodEP11CompileTask + 0xc9a 0x00007fe4a2cf2486 _ZN13CompileBroker20compiler_thread_loopEv + 0x5d6 0x00007fe4a32ae77f _ZN10JavaThread17thread_main_innerEv + 0xdf 0x00007fe4a32ae8ac _ZN10JavaThread3runEv + 0x11c 0x00007fe4a3161f78 _ZL10java_startP6Thread + 0x108 ----------------- 9528 ----------------- 0x00007fe4a30c8eaa _ZNK7MemNode20can_see_stored_valueEP4NodeP14PhaseTransform + 0x25a 0x00007fe4a30c9c74 _ZNK8LoadNode5ValueEP14PhaseTransform + 0xf4 0x00007fe4a31ac87a _ZN12PhaseIterGVN13transform_oldEP4Node + 0xea 0x00007fe4a31a9fcc _ZN12PhaseIterGVN8optimizeEv + 0x1bc 0x00007fe4a2ce51a9 _ZN7Compile8OptimizeEv + 0x159 0x00007fe4a2ce6fea _ZN7CompileC1EP5ciEnvP10C2CompilerP8ciMethodibbb + 0x13ca 0x00007fe4a2c34818 _ZN10C2Compiler14compile_methodEP5ciEnvP8ciMethodi + 0x198 0x00007fe4a2cf14da _ZN13CompileBroker25invoke_compiler_on_methodEP11CompileTask + 0xc9a 0x00007fe4a2cf2486 _ZN13CompileBroker20compiler_thread_loopEv + 0x5d6 0x00007fe4a32ae77f _ZN10JavaThread17thread_main_innerEv + 0xdf 0x00007fe4a32ae8ac _ZN10JavaThread3runEv + 0x11c 0x00007fe4a3161f78 _ZL10java_startP6Thread + 0x108 ~~~