FULL PRODUCT VERSION :
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)


FULL OS VERSION :
Linux server1.glassbox.com 2.6.10-1.770_FC3 #1 Thu Feb 24 14:00:06 EST 2005 i686 athlon i386 GNU/Linux

EXTRA RELEVANT SYSTEM CONFIGURATION :
More details are in the attached hs_error*.log files

A DESCRIPTION OF THE PROBLEM :
Our application server crashes after 1 to 30 days of operation. I don't know what is causing the crash. I believe the machine is always running the VM in -server mode. We also experienced this crash on Sun Java 1.5.0_0x VM's: we upgraded in the hopes that it would fix the error.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run server. Wait 0 to 30 days.

EXPECTED VERSUS ACTUAL BEHAVIOR :
The server doesn't crash.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
See hs_error.log. The process exits.


REPRODUCIBILITY :
This bug can be reproduced often.

CUSTOMER SUBMITTED WORKAROUND :
None. Only to restart and monitor to minimize downtime.
Posted Date : 2007-02-21 00:05:05.0

N/A

You have run out of memory during a garbage collection.  From the hs_err log:

    Heap
     def new generation   total 36288K, used 36288K [0x86ea0000, 0x89600000, 0x89600000)
      eden space 32256K, 100% used [0x86ea0000, 0x88e20000, 0x88e20000)
      from space 4032K, 100% used [0x88e20000, 0x89210000, 0x89210000)
      to   space 4032K,   0% used [0x89210000, 0x89210000, 0x89600000)
     tenured generation   total 483968K, used 483968K [0x89600000, 0xa6ea0000, 0xa6ea0000)
       the space 483968K, 100% used [0x89600000, 0xa6ea0000, 0xa6ea0000, 0xa6ea0000)
     compacting perm gen  total 108288K, used 108090K [0xa6ea0000, 0xad860000, 0xb5ea0000)
       the space 108288K,  99% used [0xa6ea0000, 0xad82e980, 0xad82ea00, 0xad860000)

The eden and from spaces are 100% used, as is every last byte in the tenured generation.  That should ordinarily throw an OutOfMemoryError at you rather than crashing, so I don't know why it crashed for you.

It would be diagnostic if you could run with a somewhat larger heap.  E.g., instead of your -Xmx512m, can you run with -Xmx768m?  You seem to have 1GB of physical memory on the machine, though you are also asking for -XX:MaxPermSize=240m, so maybe -Xmx768m is pushing it.  Even -Xmx640m might be enough to decide if that's the problem, or if it just delays the problem.  On the other hand, you are only using 105MB of the permanent generation, so maybe you could shrink that judiciously.  Or cut back on the data sets you are handling and see if that runs in -Xmx512m -XX:MaxPermSize=240m.

Running with the additional command line flags 

    -XX:+PrintGCTimeStamps -XX:+PrintGCDetails 

would give us more information on what the collector is doing.  If you could gather that output and attach it to this bug it would help.  Thanks.

Have you run any memory profilers to see if your heap has only what you think it should have in it?  Running out of memory after 30 days might be symptomatic of a "leak", i.e., unintentional retention of data structures.
Posted Date : 2007-04-09 20:21:14.0

I discovered that the server was in fact defaulting to client mode so I'm testing with -server now to see if this improves the situation.

Running with -server crashed fairly quickly on the first attempt. See also http://www.glassbox.com/jira/secure/attachment/10030/hs_err_pid840.log for the server VM crash log and http://www.glassbox.com/jira/secure/attachment/10031/hs_err_pid29367.log for the client VM crash log.

I tried running jmap to get heap information about a possible but it failed. I've now restarted with -server and a larger heap (800m) and 192m of permgen. Hopefully this will at least let me get a heap dump.