Batch calls to JVM TI RedefineClasses leaks around 200 megabytes on the redefinition of a thousand classes.

Although RedefineClasses has a serious leak in perm gen, it appears to be unrelated and completely pales in comparison to this, which is a leak in native (not heap, not perm gen) memory.

I have very  customer  confidence that the leak is in the classes_do line:

  RC_TRACE(100, ("+CLS_DO "));

  // Adjust constantpool caches and vtables for all classes
  // that reference methods of the evolved class.
  SystemDictionary::classes_do(adjust_cpool_cache_and_vtable);

  RC_TRACE(100, ("-CLS_DO "));

The trace lines correspond ro the critical portion of the attached logs, which look like this:

  (486, 21352) RedefineClasses: ++SINGL Memory: 8k page, physical 2097152k(585656k free)
  (486, 21352) RedefineClasses: +CLS_DO Memory: 8k page, physical 2097152k(585656k free)
  (486, 21352) RedefineClasses: -CLS_DO Memory: 8k page, physical 2097152k(585048k free)
  (486, 21352) RedefineClasses: --SINGL Memory: 8k page, physical 2097152k(585048k free)

for one loop through VM_RedefineClasses::redefine_single_class.  This sample (picked at random) shows ~600k leaked on the redefine of a single class.  That this is the leak location was cross checked by commenting out the classes_do line, the leakage reduced from 200mb to 2mb.  The numbers in parens are heap and perm gen respectively (in the loading phase the log shows leakage in perm gen but that is not happenning at all with this leak).

When the instrumentation is further drilled down, it seems to show leakage spread across the four calls to adjust_method_entries (logs for this not fine grained enough and not attached).  

The amount of leakage is exponential in the number of classes redefined and surprisingly only lightly correlated with the number of loaded classes --

For 1300 loaded classes:
  Redefined     Leaked
  Classes       Memory
     10            40K
    100         4,368K
   1000       196,020K

For 300 loaded classes:
  Redefined     Leaked
  Classes       Memory
     10            32K
    100         3,728K

The exponential  customer  corrresponds to the exponential performance behavior of this very same line.  Which is n-cubed by my prior analysis.

Attached are:
  redef_log      A log of the attached test run on Solaris-sparc with 1200 classes redefined
  redeftest.zip  A simple test that does the identity redefine by caching on the class load hook
  nb_redef_log   What happens when NB profiler attempts to profile itself
  hr_err_pid3440 The hs_err file when it OOME ar 2gb.

The hackery used in the logs does not deserve being committed to the record, but basically is just a call to:

   os::print_memory_info(tty);

The heap info, if the lesser problem is attacked, used:

  ResourceMark rm(Thread::current());

  // Calculate the memory usage
  size_t heap_used = 0;
  size_t non_heap_used = 0;
  
  for (int i = 0; i < MemoryService::num_memory_pools(); i++) {
    MemoryPool* pool = MemoryService::get_memory_pool(i);
    MemoryUsage u = pool->get_memory_usage();
    if (pool->is_heap()) {
      heap_used += u.used();
    } else {
      non_heap_used += u.used();
    }
  }

Have fun!
Posted Date : 2005-10-21 15:09:14.0

N/A

This is horrific and must be fixed in Mustang FCS and a Tiger update.
Posted Date : 2005-10-21 15:15:15.0

Robert has pointed out that there may be more than one memory
leak at work here. The majority of the memory leak is due to
a missing ResourceMark in:

src/share/vm/prims/jvmtiRedefineClasses.cpp:
VM_RedefineClasses::adjust_cpool_cache_and_vtable():

    {
      // the previous versions' constant pool caches may need adjustment
      PreviousVersionWalker pvw(ik);
      for (PreviousVersionInfo * pv_info = pvw.next_previous_version();
           pv_info != NULL; pv_info = pvw.next_previous_version()) {
        other_cp = pv_info->prev_constant_pool_handle();
        cp_cache = other_cp->cache();
        if (cp_cache != NULL) {
          cp_cache->adjust_method_entries(_old_methods, _new_methods);
        }
      }
    } // pvw is cleaned up

The PreviousVersionInfo objects returned via the PreviousVersionWalker
contain a GrowableArray of handles. These GrowableArrays are normally
cleaned up by the nearest ResourceMark. There is a ResourceMark in the
calling path of this code, but it doesn't seem to be doing the trick.
I added one to the beginning of VM_RedefineClasses::doit(), but that
didn't do the trick either.

It occurred to me that those GrowableArrays are useless after the
PreviousVersionWalker is destroyed so I looked at adding a
ResourceMark field in addition to the HandleMark field that I
already put in the PreviousVersionWalker. Can't do that because
the GrowableArray code says when you have a GrowableArray of
handles, you have to destroy the handles before the GrowableArray.
I don't think there is a way to insure the order in which field
destructors are called so we have to add ResourceMark objects
to all the places where PreviousVersionWalker objects are created.

ETA: It is probably safe to add the ResourceMark field to
PreviousVersionWalker because the warning in the GrowableArray
code is for the situation where the HandleMark frees the handles
and the GrowableArray is still around to access those now stale
handles. With PreviousVersionWalker, the intention is to destroy
both fields since we are done with them and there is no other
way to access to GrowableArrays.

So while it should be safe to add the ResourceMark field, it
doesn't work because it requires that instanceKlass.hpp depend
on resourceArea.hpp and that causes an include loop. I would
have to move the PreviousVersionWalker stuff to another file
and I don't want to do that.

ETA: Adding a ResourceMark to the top of VM_RedefineClasses::doit()
didn't work. Adding one just before the loop that calls
redefine_single_class() doesn't do it either. Adding one in the
loop just before the call to redefine_single_class() does the
trick. That doesn't make any sense, but...
Posted Date : 2005-11-09 22:43:56.0

You learn something new everyday. instanceKlass::itable() and
instanceKlass::vtable() allocate wrapper objects every time
they are called. So when adjust_cpool_cache_and_vtable() calls
them a wrapper object is leaked for each call. vtable() and
itable() are called at most once per call to
adjust_cpool_cache_and_vtable(), but adjust_cpool_cache_and_vtable()
is called alot so we have a decent sized leak.
Posted Date : 2005-11-08 22:31:35.0

An additional perspective (via observation of the Java process from the OS, process stats provided)  when instrumenting JBoss server is available here:

http://www.netbeans.org/issues/show_bug.cgi?id=61994

Will this ever be fixed in the 1.5 series as well?

According to the release notes for 1.5_08 it is fixed there also