United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: 7074579 G1: JVM crash with JDK7 running ATG CRMDemo Fusion App
7074579 : G1: JVM crash with JDK7 running ATG CRMDemo Fusion App

Details
Type:
Bug
Submit Date:
2011-08-03
Status:
Closed
Updated Date:
2012-01-10
Project Name:
JDK
Resolved Date:
2011-09-30
Component:
hotspot
OS:
linux_oracle_5.4
Sub-Component:
gc
CPU:
sparc
Priority:
P2
Resolution:
Fixed
Affected Versions:
7
Fixed Versions:
hs22

Related Reports
Backport:

Sub Tasks

Description
We are evaluating G1GC for a Fusion App workload ATG CRMDemo. After around 10-15 mins into the load test we are seeing a JVM crash. We tried with both the JDK7b147 and JDK7b146 and observing the crash. I am attaching a gz file to this bug which has the GC log, hs_err file and hsperfdata file. We are  running with the following JVM Args :

-server -Xms2560m -Xmx2560m -Xmn512m -XX:+UseLargePages -XX:+AggressiveOpts -XX:-DoEscapeAnalysis -XX:PermSize=384m -XX:MaxPermSize=384m -XX:ReservedCodeCacheSize=64m

Here is the server crash output the Weblogic server log. I also have a core file in case if you want to look at it. Please let me know in case if you need any thing else.

================
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002b9e80fd2025, pid=19868, tid=1124489536
#
# JRE version: 7.0-b147
# Java VM: Java HotSpot(TM) 64-Bit Server VM (21.0-b17 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x6ea025]  oopFactory::new_objArray(klassOopDesc*, int, Thread*)+0x25
# 
# Core dump written. Default location: /export/home/bench/RC7/Oracle/Middleware/user_projects/domains/atg_domain/core or core.19868
# 
# An error report file with more information is saved as:
# /export/home/bench/RC7/Oracle/Middleware/user_projects/domains/atg_domain/hs_err_pid19868.log
# 
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# 
=================================

                                    

Comments
EVALUATION

See main CR
                                     
2011-09-12
EVALUATION

http://hg.openjdk.java.net/hsx/hotspot-main/hotspot/rev/87e40b34bc2b
                                     
2011-08-17
EVALUATION

http://hg.openjdk.java.net/hsx/hotspot-rt/hotspot/rev/87e40b34bc2b
                                     
2011-08-17
EVALUATION

http://hg.openjdk.java.net/hsx/hotspot-gc/hotspot/rev/87e40b34bc2b
                                     
2011-08-11
SUGGESTED FIX

diff -r 0defeba52583 src/share/vm/services/gcNotifier.cpp
--- a/src/share/vm/services/gcNotifier.cpp      Tue Jul 12 16:32:25 2011 -0700
+++ b/src/share/vm/services/gcNotifier.cpp      Fri Aug 05 10:32:12 2011 -0700
@@ -100,9 +100,14 @@
   // Fill the arrays of MemoryUsage objects with before and after GC
   // per pool memory usage

-  klassOop muKlass = Management::java_lang_management_MemoryUsage_klass(CHECK_NH);   objArrayOop bu = oopFactory::new_objArray( muKlass,MemoryService::num_memory_pools(), CHECK_NH);
+  klassOop mu_klass = Management::java_lang_management_MemoryUsage_klass(CHECK_NH);
+  instanceKlassHandle mu_kh(THREAD, mu_klass);
+  objArrayOop bu = oopFactory::new_objArray(mu_kh(), MemoryService::num_memory_pools(), CHECK_NH);
   objArrayHandle usage_before_gc_ah(THREAD, bu);
-  objArrayOop au = oopFactory::new_objArray(muKlass,MemoryService::num_memory_pools(), CHECK_NH);
+  objArrayOop au = oopFactory::new_objArray(mu_kh(), MemoryService::num_memory_pools(), CHECK_NH);
   objArrayHandle usage_after_gc_ah(THREAD, au);

   for (int i = 0; i < MemoryService::num_memory_pools(); i++) {
                                     
2011-08-05
EVALUATION

Potential naked oop across a call that could potentially GC in createGCInfo in gcNotifier.cpp:

static Handle createGcInfo(GCMemoryManager *gcManager, GCStatInfo *gcStatInfo,TRAPS) {

  // Fill the arrays of MemoryUsage objects with before and after GC
  // per pool memory usage

  klassOop muKlass = Management::java_lang_management_MemoryUsage_klass(CHECK_NH);
  objArrayOop bu = oopFactory::new_objArray( muKlass,MemoryService::num_memory_pools(), CHECK_NH);
  objArrayHandle usage_before_gc_ah(THREAD, bu);
  objArrayOop au = oopFactory::new_objArray(muKlass,MemoryService::num_memory_pools(), CHECK_NH);
  objArrayHandle usage_after_gc_ah(THREAD, au);

Suppose the call to new_objArray results in a GC then the class to which muKlass points may have been moved.

Note: the ServiceThread is a Java thread. Even though it is handling a gc notification - we are not at a safepoint when executing the above code:

During a GG, the VM thread will push the gc notification event and signal the Service_Lock. The ServiceThread, which was blocked waiting on the Service_Lock, is still stopped (for the safepoint). _After_ the safepoint the ServiceThread restarts; obtains the Service_Lock, and starts procssing the notification and, when the first allocation above is executed, causes a GC.
                                     
2011-08-05



Hardware and Software, Engineered to Work Together