When Weblogic Server (WLS) loads more than ~400 EJBs, the HotSpot VM throws an OutOfMemoryError exception. This occurs even though there appears to be lots of memory available; in our scenario, there was over 460 mb of memory available
based on a call to Runtime.getRuntime().freeMemory(). Loading large number
of EJBs was possible with previous versions of the VM.

The HotSpot VM was throwing the OutOfMemoryError when trying to allocate
PermGeneration space. It appears that the HotSpot VM has different sections
of memory. The permanent generation section (used for classes, methods, 
symbols ??) has an initial size of 1 mb and a max size of 32 mb.

Fortunately, this limit can be raised at runtime using the

        java -server  -XX:MaxPermSize=65536K

command line argument. The above command raises the MaxPermSize to 64 mb. With this limit, we were able to load additional ejbs.

However, the -XX options are not supported, and ISVs/Customers are wary about using XX options.

==============
Note that currently, increasing the MaxPermSize only delays the failure.

  xxxxx@xxxxx   2001-05-22

Use the -client VM parameter instead of -server.

Increase the MaxPermSize value by using the command line flag:
 -XX:MaxPermSize=128M  

See http://java.sun.com/docs/hotspot/VMOptions.html

Changing to RFE from bug as the customer issue is a request to stabilize /
make official the current -XX option which yields the desired result.
  xxxxx@xxxxx   2001-02-16

Changing from RFE to Bug because throwing an OutOfMemoryError when we are
not out of memory is not correct. The fact that the customer suggests a
particular resolution to the problem (which indidentally does not resolve 
the problem, only defers it) doesn't make it anything less than a bug, and
a serious one at that.

  xxxxx@xxxxx   2001-04-19

Try as a workaround using the -client VM parameter instead of -server.

  xxxxx@xxxxx   2001-05-22

I need a testcase or access to the application that is exhibiting this problem.
There have been many reports of OOM but no testcase to work with. I will
be closing this bug next week if I don't hear anything.
  xxxxx@xxxxx   2001-08-24

This problem can be reproduced consistently by running
the J2EE compatibility test suite (CTS) aginst J2EE RI 1.3
You will need to contact   xxxxx@xxxxx   (manager)
and   xxxxx@xxxxx   (engr) for more information
on how to setup and run CTS. You can also look at the
bug 4412393. We workaround the problem by increasing
the MaxPermSize manually.
  xxxxx@xxxxx   2001-08-24

Please do not close this bug until we've checked for the same problem in 1.4.
If a reproducable test case is found, please forward to me!

  xxxxx@xxxxx   2002-02-18

Please see the workaround suggested. You have to tune the -XX:MaxPermSize value
for your particular application. By default for server compiler its 64M and
32M for the client compiler.
  xxxxx@xxxxx   2002-05-29

We encountered similar problem too. Sometimes JVM crashed
with or without OutOfMemory exception. Crashing is not
acceptable, you know.

Increasing the MaxPermSize means that my JVM will crash later instead of sooner.  Which means I cannot 
run this in a production site that is not supposed to crash.  Is Sun planning on fixing this?

mreiche@ea.com

We're getting the same thing, although we're not seeing OutOfMemoryError, the JVM just core dumps and 
crashes and burns

WebLogic 5.1 sp6 running on Solaris 2.6
JDK 1.3

#
# HotSpot Virtual Machine Error, Unexpected Signal 11
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Error ID: 4F533F534F4C415249530E435050079A 01
#
# Problematic Thread: prio=5 tid=0x1049b0 nid=0xb runnable 
#

This worked fine in our development and staging environments for weeks. One hour after we released to 
production the HotSpot JVM core dumped. This is simply unacceptable....we've gone back to JDK1.2.1_04 
which is without a doubt the most stable JVM out there. 1.2.2_05 has bugs that JSPs hit.

Very disheartening....

This bug affect all jvm 1.3 (Linux, nt, ...) or only the 
solaris version ?

I've got crash on Win2000, JDK 1.3 under the WLS load.

WebLogic support tells me a fix is on the way...
Mike.
===================
FR: ytian

CASE_ID_NUM: 205156
MESSAGE:
Michael,

Here is an update from SUN about the problem you are seeing:

>>I had filed the bug (4390238) and escalation on behalf of BEA Weblogic. The fix
>>is in progress, and I'll have the build next week. The fix will be integrated
>>into the next point release of JDK1.3.

>>WOrkaround use -XXMaxPermSize=65536K

Hope we can get it resolved soon.

Thanks

Yu Tian

Our application is a stand alone desktop application using 
jdk 1.3 as a client.Swing components are used as an 
UserInterface on NT operating system.Similar problem occurs 
but no using weblogic.I increased -Xmx option to maximum of 
200m ,but that does'nt help in going outofmeory 
error.Jprobe and optimzeit did not prove the memory leaks 
in the code.I suspect hotspot jvm not managing the 
garbagecollection appropriatly.
please let me know if anybody had work around to solve this 
problem.

Looks like it is still broken in 1.3.1

Why is this an RFE?  It's broken. With -XX it still crashes - only a little later.
 Even with -XX:MaxPermSize=512m

Still broken in 1.3.0_02 (on Solaris at least).

Is it fixed in 1.3.0 03.  Any workaround to prevent the
crash?

We're seeing this condition on Linux with JDK 1.3.1-b24 and 
WL 6.0sp1.  Help Sun we need a fix here!

Since the bug is still in JDK 1.3.1, and has been open for 
so long, one can only assume it requires a major fix in the 
JVM. So do we have to wait for JDK 1.4?

I find it sad that Weblogic 6.0SP1 w/ JDK 1.3.1 has this 
problem.  BEA can't even get Sun to fix this?

I guess no one uses EJB's anymore now that the hype is 
gone.  On to Web Services.... :(

Does anyone know if the outofmemory error is just related 
to an initial configuration taking into acount the maximum 
number of classes that will be loaded, or if there is also 
a memory leak problem. i.e. even though all classes are 
loaded initially without an error, will you eventually run 
out of memory because of a memory leak.

I had a similar situation where we tested throughly, though 
never for long enough. About one day after launching to 
production we started getting this. Now I am scared to go 
to sleep for more than two hours at a time so I can keep an 
eye on things. I pushed hard to get my organization to go 
to an all Solaris, all Java environment and they did and 
now I look like a fool. I get the same errors using -client 
or -server. I hope I am still employed by the time a fix 
comes out.

The woraround says, "Use the -client VM parameter instead 
of -server."
Has anyone checked whether the -client or -hotspot option 
solve this problem ?
It seems -XXMaxPermSize=65536K does not solve always. Does 
it ?

I am now using IBM JRE and the problem does not show. I 
have performed stress testing and the problem seems to go 
away. Also runing the Sun JRE in classic mode, makes the 
problem go away, albiet with low performance.

Ya me to faced the same prob when trying to run an 
application which does simswap ,after six or seven times 
running the application it throws out of memory error

This bug has nothing to do with the memory setings -Xms or -
Xmx... The hot-spot does not launch the gc on complex 
threads applications because the gc is very low low 
priority task... the hot-spot will reach outofmemory error 
before it can run the gc or runFinaliztion... the only way 
to avoid this situation is Sun creates a new 
method/instruction that forces the gc and/or runFinaliztion 
as priority one... and I will like another method to 
disable/enable the gc on runtime...  meanwhile I have to 
live with the creation of an artificial thread set as high 
priority to be able to force the gc and
         Thread.currentThread().setPriority
	  (Thread.MAX_PRIORITY);
            System.gc();
            System.runFinalization();

I am getting this with WLS 6.1 SP 1 and JDK 1.3.1_03.  
Please fix this.

We also encountered this issue.
And I realized that the problem only occured on non-SMP 
machine.
We never had this issue on SMP-system.
So I think there is a string attached in threding mechanism.
Any suggestions?

   we also meet this programe with "out of memory".And 
after we set the MaxPermSize the problem is solved.But The 
Ejbs will get ClassCastException When the ejb use for so 
days.Util we redeploy the Ejb,the ejbs get right again,We 
use weblogic6.0 sp2+jdk1.3+sun solaris 7.
   The bea support said that is a bug of jdk 1.3 for 
solaris.
   please fix the bug.otherwise the Ejb counld not be use.

We are having core-dump problem on Weblogic 6.0SP1. Using 
1.3.1 on HP-UX. Our roll-out date has to be postponed 
because of this. If this continues than no way we can ever 
roll-out and our multi-million project need to be scraped. 
Probably need to sue Sun Microsystems.

You can increase the server virtual memory, make a cold 
restart of the server..

I am having the save error ID and core dump in an solaris + 
weblogic (jdk1.3.0) env. 
I tried the -client option, but core dump
still occurs every 30 minutes or so.
It is really a shame that Sun is just not able to fix a 
critical bug such as this.  Words from xxxxx@xxxxx 2001-05-
22 just makes no sense. Don't close the bug because you 
don't have a testcase, could you please just write one?

I'm voting because Mark G told me to

We are seeing this issue crop up when too many jsps are 
loaded into our solaris 1.3 jvm.  To REPLICATE THIS, try 
creating about 4000 or so dummy jsps and write a script to 
load these in solaris (with weblogic, hit the weblogic 
instance).  You will see the jvm DIE.

What a crappy bug

We saw this bug (or very similar consequences) with jdk 
1.3.1 b24 and 01 on Sun single CPU E220R Solaris 8. Our BEA 
Weblogic server 6.1sp1 crashed without warnings or 
messages. We saw the OutOfMemory error just a few times.
We tried MaxPermSize at 128m and 256m without success 
except delaying the crash. We increased the number of 
threads allowed in weblogic and used a patch given by BEA 
support. Now our server is up for more than one week. But 
did we solve the problem or just delayed it ? 
As we can't trust our platform anymore, we need to buy more 
servers and more CPU and more weblogic licence to double 
every component. This bug seems to be very lucrative for 
everyone but us !

Truly a bizarre situation. We were contemplating moving our 
entire company over to Java, Weblogic and Solaris 
environment, but now I am having serious doubts about Sun 
Microsystems ability to deal with serious problems. 
Microsoft may be bug riddled but at least they respond 
positively and enthusiastically to problems which occur! 
Sun wakeup!

Sometimes JVM crashed with or without OutOfMemory 
exception. 

I have similar problem, creating a huge image or
double[][][] even when there was enough space if gc() were run.

WORKAROUND: call System.gc() manualy before making a huge
number of objects (even if System.gc() automatically should
be called anyway durring the object creation). It doesn't
seem to execute while creating many objects at once.

Another problem is that this code will throw an out of
memory exception
int i = [huge int]
Object[] oa = new Object[i];
try {
  oa = new Object[i];
} catch(Throwable t) {
  System.out.println("didn't work");
}
for some value of i (or of -Xmx). 

However, this code will work for that value of i
int i = [same huge int]
Object[] oa = new Object[i];
oa = null;
try {
  oa = new Object[i];
} catch(Throwable t) {
  System.out.println("shouldn't get here");
}
 for me (default or -Xmx64m) i=10000000 works.

I'm still getting the same crash.  It happened to both 
single and dual cpu machines, both running Solaris 2.8.

It happened to JRE1.3.1 and JRE1.3.1_02.

Please suggest a viable work around!  We can't ship without
a stable JVM!!!!


#
# HotSpot Virtual Machine Error, Internal Error
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Error ID: 4349254E560E4350500090 01
#
# Problematic Thread: prio=5 tid=0xa6910 nid=0xb runnable
#

We encountered similar problem too. Sometimes JVM crashed
with or without OutOfMemory exception. Crashing is not
acceptable

Looks like the -XX:NewSize=4m fixed the problem. I read the 
article at 
http://developer.java.sun.com/developer/TechTips/2000/tt1222
.html which talks about the different tuning options. If I 
use -Xmx64m and -XX:NewSize=4m options, the memory usage is 
not going up at all. Not sure if the "nursery GC" has a 
memory leak or it is just running too frequent that the 
Full GC never had a change to kick in. I'll tomorrow for 
sure if it fixed my problem.

Got this problem too on Weblogic SP9 and java 
version "1.3.1_02"
Java(TM) 2 Runtime Environment, Standard Edition (build 
1.3.1_02-b02)
Java HotSpot(TM) Server VM (build 1.3.1_02-b02, mixed mode)

running in a cluster

Is there a way to workaround this by coding differently?

What if request scope is used instead of e.g. session scope 
for useBean's in JSP's

Would the objects be garbage collected more often, or just 
fill up memory faster?

Well, here I am.  We're running latest Tomcat Solaris 2.6,  
JDK 1.3.1 (1.3.1-b24, mixed mode).  For the past two days - 
both of which when critical demos are going on - Catalina 
has bombed with this error, apparently.  Here's the log

#
# HotSpot Virtual Machine Error, Unexpected Signal 10
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Error ID: 4F533F534F4C415249530E43505007A6 01
#
# Problematic Thread: prio=5 tid=0x4b9fc8 nid=0x2d runnable 

We encounter the same problem. Weblogic 6.1sp2 with  jdk 1.3.1_01 on Solaris 8 crashes without reporting  
any OutOfMemory exception during parsing an XML feed. This happens about all 3 to 12 hours which just is 
totaly inacceptable!

To duplicate this bug, you need to generate lots of 
classes. Try this:

package permgeneration;

import java.lang.reflect.*;

public class Main {
    static InvocationHandler invocationHandler = new 
InvocationHandler() {
        public Object invoke(Object o, Method m, Object[] 
args) {
            return null;
        }
    };

    public static void main(String[] av) throws Exception {
        java.util.Collection clt = new java.util.ArrayList
();

        for(int b = 1; b< 1<<16; b++) {
            int bitCount = 0;
            for(int i = 0;i<16; i++) {
                if((b & (1<<i))!=0) {
                    bitCount++;
                }
            }
            Class[] clazzV = new Class[bitCount];

            int atBit = 0;
            for(int i = 0;i<16; i++) {
                if((b & (1<<i))!=0) {
                    clazzV[atBit++] = Class.forName
("permgeneration.I" + i);
                }
            }

            System.out.print(b);
            System.out.print(" - ");
            System.out.print(Runtime.getRuntime().freeMemory
());
            System.out.print(" - ");
            Object p = Proxy.newProxyInstance
(Main.class.getClassLoader(), clazzV, invocationHandler);
            System.out.print(p.getClass().getName());
            System.out.print(" - ");
            clt.add(p);
            System.out.println(Runtime.getRuntime
().freeMemory());
        }
    }
}

interface I0  { void i00();void i01();void i02();void i03
();void iq4();void iq5();void iq6();void iq7();}
interface I1  { void i01();void i11();void i12();void i13
();void iw4();void iw5();void iw6();void iw7();}
interface I2  { void i02();void i21();void i32();void i23
();void ie4();void ie5();void ie6();void ie7();}
interface I3  { void i03();void i31();void i42();void i33
();void ir4();void ir5();void ir6();void ir7();}
interface I4  { void i04();void i41();void i52();void i43
();void it4();void it5();void it6();void it7();}
interface I5  { void i05();void i51();void i62();void i53
();void iy4();void iy5();void iy6();void iy7();}
interface I6  { void i06();void i61();void i72();void i63
();void iu4();void iu5();void iu6();void iu7();}
interface I7  { void i07();void i71();void i82();void i73
();void ii4();void ii5();void ii6();void ii7();}
interface I8  { void i08();void i81();void i92();void i83
();void io4();void io5();void io6();void io7();}
interface I9  { void i09();void i91();void iq2();void i93
();void ip4();void ip5();void ip6();void ip7();}
interface I10 { void i10();void iq1();void iw2();void iq3
();void ia4();void ia5();void ia6();void ia7();}
interface I11 { void i11();void iw1();void ie2();void iw3
();void is4();void is5();void is6();void is7();}
interface I12 { void i12();void ie1();void ir2();void ie3
();void id4();void id5();void id6();void id7();}
interface I13 { void i13();void ir1();void it2();void ir3
();void if4();void if5();void if6();void if7();}
interface I14 { void i14();void it1();void iy2();void it3
();void ig4();void ig5();void ig6();void ig7();}
interface I15 { void i15();void iy1();void iu2();void iy3
();void ih4();void ih5();void ih6();void ih7();}

thanks pmurray your suggestions worked out.

We encountered similar problem

4390238

I can run 700+ EJB's on WL6.0 sp2 by compiling the stub 
classes (using weblogic.ejbc) with IBM's javac and then 
running the resulting jar on weblogic under SUN's JRE ! Ha.

Does upgrading to J2SE 1.4.0 solve the problem? Merlin has a much improved GC 
implementation.  
 
Do we really need to make fixing a bug in an obsolete version of Java the 
highest priority, as indicated by its #1 ranking in the top 25 bugs right now? 
 
I would much rather see Sun's effort put into current and future J2SE 
releases. 

No, fix both 1.3 and 1.4. I bet a lot of customer will 
still continue to use 1.3 for a long time.

Hi is there any stable workaround or solution. We 
desparately need our application working. Can someone 
summarize.

regards,
Ardaman

krud

I am experiencing the same problem in my project where i 
use custom tags in a web application for 
internationalization.So please do not close the bug. My 
project will be the ideal test case. Try running a page 
with more than 150+ custom tags in it.

Not reproducible???!  The suggested workaround only delays 
the problem instead of fixing it.  This is totally 
unacceptable for J2EE environments.  It _is_ reproducible 
using the methods suggested above.  We have to restart our 
J2EE server every couple of days because of this bug.  I'm 
very surprised such an important bug gets such sloppy 
attention.

How can this bug be closed without giving a fix? There are 
enough comments for reproducing this problem.

CLOSED?? NOT REPRODUCIBLE??? Hey, Sun -- what's going on??
are you tired from fixing bugs? want all the heavy
applications to move to .NET?? what we need now - STABILITY!!

"closed, not reproducible" - shame on Sun. :(

I agree it is hard to believe even without using Weblogic I
can duplicate this bug just loading all of the classes in
our app. Around 15,000 classes. Or under Weblogic create a
single jar that contains the ejb-jar.xml that contains 400 EJBs.

I just don't believe it !!!. Not reproducable !!!!
For all the others: I discover that the problem does not 
exists in the IBM JDK 1.3.x

Unbelievable.

56 mails from people with this problem, and SUN says that 
is "not reproducible". 

Maybe do you think we are lying?

Sun sucks !!

Is the number (400) mentioned above the number of beans 
deployed or they the total number of instances of the 
beans.For e.g minimum pool size is 50 and the beans deploted 
are 100,then number  of instance will be 5000..Would this 
create a problem????....Please reply to my mail id at 
shwetas@snb.psi.soft.net

Did anyone /anyone find a workaround

- kamy
(kmantha@btmna.com)

-XX:MaxPermSize=128m ...is this a stable workaround...

if anyone has solutions for this....please do let me know 
mshah@mortgagehub.com

1.3.1 -> 1.3.0  worked.

We encountered the same problem.In a session bean we get 
a Blob from Database and fill a byte array. this Byte array is 
returned to the EJB client. Irony is that the outofMemory 
remote exception occures at client.The Server continues to 
run and any other process(Not thread) can access the bean 
without an error.We are using IPAS 3.0 app server.

problem persists with 1.3.1_04 on Sparc Solaris 8

We have the same problem. The machine has 8G mem, 4G
allocated to Java and everything goes to hell after 400M
have been used by JSPs. 

What expects SUN us to do? What are we buying incredibly
expensive Sun machines for if even their own JDK doesn't run
on it in enterprise scale applications. 

This problem IS fixed in 1.4.  It was closed as not
reproducible because
it could not be reproduced in 1.4 (and 1.3 will not be patched)

This currently happens to our web application running on
iPlanet (Sun One) 4.1 web server using JDK 1.3.1_05.  The
heap and GC look fine (and available memory), but every
other day or so, the JVM crashes with OutofMemory errors and
the server is set to auto-restart.  I cannot believe Sun is
going to close this when it is so apparent.  Oh wait, yes I
can, considering we spent a year trying to fix a memory leak
in our code and it turns out one of their service packs
fixed it.  Imagine that!

We had the same problem while testing WLS 6.1 SP 3 on 
Solaris 2.8. We could reproduce the problem simply with a 
stress test (using Grinder). If you do a stress test that 
solicite always one simple JSP (hello world) you'll see in the 
WLS console that the memory is consumed, tha GC don't free 
all the memory until the OutOfMemory error occurs.

The problem could be reproduced in 10 minutes.

Using the -XX:MaxPermSize looks like solves the problem.

The bug is unfortunately still there in jre v131_06 on Linux. 

java version "1.3.1_06"
Java(TM) 2 Runtime Environment, Standard Edition (build
1.3.1_06-b01)
Java HotSpot(TM) Client VM (build 1.3.1_06-b01, mixed mode)

Actually, I am running JDK 1.3.1_04 on Solaris, and -
XXMaxPermSize=128M is set, and the problem is still 
occuring during intensive load testing......

This problem is also happening on WAS V 4.0.2 (IBM 
WebSphere Application Server) even when the value of -
XX:MaxPermSize=128M, this happens during load testing 
with 400 co-current users.  Should we increase this value to 
256M??, I know this does not solve the porblem, but delays 
it....

www

This is reproducible with JDK1.4.1 . I am deploying 700 beans 
to OC4J 9.0.3 .. When my client application tries to connect, 
OC4J gives OutOfMemoryError.. After setting MaxPermSize to 
128m, the problem is solved..

-XX:MaxPermSize=256m seems to work for now. We certainly 
cant go live with our trading system until we have an answer 
from Sun as to whether this has simply deferred the problem. 
I can also reproduce it in using WLS 6.1 on 1.4.1.

try the feature out

I reproduce the same OutOfMemoryError after just repeat 
redeploy application about 15 times, the application has 700 
EJB, I configured 2.5G memory. At the time of crash, the 
memory usage is only around 500M.... I am really worried, 
this supposed to be factory 7x24 application. 

I feel that loading too many EJB's is causing this problem. My 
explanation is as follows:
There are too many objects being created. As a result of 
which this is error is occuring. When The EJB loaded ( I 
worked on Weblogic Server) it creates a initial pool of 1000. 
This can be reduced in the descriptor file. Try to reduce the 
poo, capacity for each component and then try to deploy 
them. I myself am working on this. And am not sure if this will 
work. You must try.

I reproduce the same OutOfMemoryError after just repeat 
redeploy application

Could any one confirm if the problem has been fixed in 
JDK 1.4.* or not please?

Just ran into this problem also.  My sincere thanks to all
those who've spent hours tracking this down in the past &
wrote up the workaround.

I am also getting this error.
xxxxxxxxxxxxxxxxxxxxxxxxxxx
Start server side stack trace:
java.lang.OutOfMemoryError: 
Start server side stack trace:
java.lang.OutOfMemoryError
	<<no stack trace available>>
End  server side stack trace
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I am using 
MEM_ARGS="-Xms128m -Xmx768m -
XX:MaxPermSize=128m"

Still getting the same problem.

I am also getting this error.
xxxxxxxxxxxxxxxxxxxxxxxxxxx
Start server side stack trace:
java.lang.OutOfMemoryError: 
Start server side stack trace:
java.lang.OutOfMemoryError
	<<no stack trace available>>
End  server side stack trace
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I am using 
MEM_ARGS="-Xms128m -Xmx768m -
XX:MaxPermSize=128m"

Still getting the same problem.

Am using Weblogic 8.1 SP4 with jdk 1.4 on solaris 9.
The server works fine when started but soon starts throwing the following errors:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 <Error> <Cluster> <evoapp02> <ManagedServer21> <ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'> <<WLS Kernel>> <> <BEA-000110> <Multicast socket receive error: java.lang.OutOfMemoryError
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

So JDK 1.4 necessarily fix the problem.