EVALUATION

ChangeSet=http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c385bf94cfb8,ChangeRequest=6935839
                                     

SUGGESTED FIX

When draining the objarray stack/queue, process a single array at a time, instead of all of them.  Par compact should also avoid keeping very large stacks around for the lifetime of the JVM; that will be done as part of 6423256: GC stacks should use a better data structure.
                                     

EVALUATION

The failing test ran out of c heap.  The additional heap usage is due to deeper marking stacks as a result of the fix for 4396719, which changed processing of object arrays to store in-progress arrays on stacks managed by the jvm instead of using recursion.  The fix was overly aggressive in pushing elements onto the marking stack, thus increasing their depth when marking large arrays.  The problem is compounded in the parallel compacting collector because the various stacks used by each GC worker are kept for the lifetime of the JVM--once a marking stack grows large, the space is never released until the VM exits.  (The serial mark-compact collector allocates and deallocates the stacks at each GC.)  On a many-core machine there will be many gc worker threads, resulting in a lot of unused space.
                                     

SUGGESTED FIX

diff --git a/src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp b/src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp
--- a/src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp
+++ b/src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp
@@ -217,21 +217,21 @@
 void ParCompactionManager::follow_marking_stacks() {
   do {
     // Drain the overflow stack first, to allow stealing from the marking stack.
+    oop obj;
     while (!overflow_stack()->is_empty()) {
       overflow_stack()->pop()->follow_contents(this);
     }
-    oop obj;
     while (marking_stack()->pop_local(obj)) {
       obj->follow_contents(this);
     }
 
+    // Process ObjArrays one at a time to avoid marking stack bloat.
     ObjArrayTask task;
-    while (!_objarray_overflow_stack->is_empty()) {
+    if (!_objarray_overflow_stack->is_empty()) {
       task = _objarray_overflow_stack->pop();
       objArrayKlass* const k = (objArrayKlass*)task.obj()->blueprint();
       k->oop_follow_contents(this, task.obj(), task.index());
-    }
-    while (_objarray_queue.pop_local(task)) {
+    } else if (_objarray_queue.pop_local(task)) {
       objArrayKlass* const k = (objArrayKlass*)task.obj()->blueprint();
       k->oop_follow_contents(this, task.obj(), task.index());
     }
diff --git a/src/share/vm/gc_implementation/shared/markSweep.cpp b/src/share/vm/gc_implementation/shared/markSweep.cpp
--- a/src/share/vm/gc_implementation/shared/markSweep.cpp
+++ b/src/share/vm/gc_implementation/shared/markSweep.cpp
@@ -111,7 +111,8 @@
       assert (obj->is_gc_marked(), "p must be marked");
       obj->follow_contents();
     }
-    while (!_objarray_stack->is_empty()) {
+    // Process ObjArrays one at a time to avoid marking stack bloat.
+    if (!_objarray_stack->is_empty()) {
       ObjArrayTask task = _objarray_stack->pop();
       objArrayKlass* const k = (objArrayKlass*)task.obj()->blueprint();
       k->oop_follow_contents(task.obj(), task.index());