xxxxx@xxxxx   2002-06-26

During the last month messages displaying "out of memory" have become more common. I've seen this a few times while running the JCK harness with jdk1.3.1_04. (while testing the 1.4.0_01 jdk) I initially thought this was a memory resource problem with the machine but I think it may be a more serious issue with the jdk.

I've seen it on both Linux and XP Pro.
I've had feedback from other engineers saying that they have also experienced a similar memory problem while using the jdk1.4.0_01 jdk to run tests.

The errors normally start appearing after a few thousand tests have been executed.

Included are examples from linux and winxp. :


Windows XP :

C:\>cd jck13^M
^M
C:\jck13>set CLASSPATH=z:/jck/JCK-runtime-13a/javatest.jar;z:/jck/JCK-runtime-13
^M
a/classes^M
^M
C:\jck13>ksh LiM.ksh^M
^M
C:\jck13>Initial Naming Context:^M
IOR:000000000000002849444c3a6f6d672e6f72672f436f734e616d696e672f4e616d696e67436f
^M
6e746578743a312e300000000001000000000000005c000101000000000f3132392e3135362e3233
^M
332e37330000040e000000000019afabcaff00000002bd04637d0000000800000000000000000100
^M
00000000000100000001000000140000000000010020000000000001010000000000^M
TransientNameServer: setting port for initial  customer  references to: 9876^M
Ready.^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
Exception occurred during event dispatching:^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
java.lang.OutOfMemoryError^M
        <<no stack trace available>>^M
Exception occurred during event dispatching:^M
An unexpected exception has been detected in native code outside the VM.^M
Unexpected Signal : EXCEPTION_ACCESS_VIOLATION occurred at PC=0x6d0d58d3^M
Function name=JNI_OnLoad^M
Library=C:\jdk1.3.1_04\jre\bin\awt.dll^M
^M
Current Java thread:^M
        at sun.awt.windows.WToolkit.eventLoop(Native Method)^M
        at sun.awt.windows.WToolkit.run(WToolkit.java:183)^M
        at java.lang.Thread.run(Thread.java:479)^M
^M
Dynamic libraries:^M
0x00400000 - 0x00405000         C:\jdk1.3.1_04\bin\java.EXE^M
0x77F50000 - 0x77FF9000         C:\WINDOWS\System32\ntdll.dll^M
0x77E60000 - 0x77F45000         C:\WINDOWS\system32\kernel32.dll^M
0x77DD0000 - 0x77E5B000         C:\WINDOWS\system32\ADVAPI32.dll^M
0x77CC0000 - 0x77D35000         C:\WINDOWS\system32\RPCRT4.dll^M
0x77C10000 - 0x77C63000         C:\WINDOWS\system32\MSVCRT.dll^M
0x6D420000 - 0x6D4F7000         C:\jdk1.3.1_04\jre\bin\hotspot\jvm.dll^M
0x77D40000 - 0x77DCD000         C:\WINDOWS\system32\USER32.dll^M
0x77C70000 - 0x77CB0000         C:\WINDOWS\system32\GDI32.dll^M
0x76B40000 - 0x76B6C000         C:\WINDOWS\System32\WINMM.dll^M
0x6D220000 - 0x6D227000         C:\jdk1.3.1_04\jre\bin\hpi.dll^M
0x6D3B0000 - 0x6D3BD000         C:\jdk1.3.1_04\jre\bin\verify.dll^M
0x6D250000 - 0x6D266000         C:\jdk1.3.1_04\jre\bin\java.dll^M
0x6D3C0000 - 0x6D3CD000         C:\jdk1.3.1_04\jre\bin\zip.dll^M
0x6D020000 - 0x6D12A000         C:\jdk1.3.1_04\jre\bin\awt.dll^M
0x73000000 - 0x73023000         C:\WINDOWS\System32\WINSPOOL.DRV^M
0x76390000 - 0x763AA000         C:\WINDOWS\System32\IMM32.dll^M
0x771B0000 - 0x772CA000         C:\WINDOWS\system32\ole32.dll^M
0x6D1E0000 - 0x6D21B000         C:\jdk1.3.1_04\jre\bin\fontmanager.dll^M
0x6D340000 - 0x6D348000         C:\jdk1.3.1_04\jre\bin\net.dll^M
0x71AD0000 - 0x71AD8000         C:\WINDOWS\System32\WSOCK32.dll^M
0x71AB0000 - 0x71AC5000         C:\WINDOWS\System32\WS2_32.dll^M
0x71AA0000 - 0x71AA8000         C:\WINDOWS\System32\WS2HELP.dll^M
0x71A50000 - 0x71A8B000         C:\WINDOWS\system32\mswsock.dll^M
0x71A90000 - 0x71A98000         C:\WINDOWS\System32\wshtcpip.dll^M
0x76F20000 - 0x76F45000         C:\WINDOWS\System32\DNSAPI.dll^M
0x76FB0000 - 0x76FB7000         C:\WINDOWS\System32\winrnr.dll^M
0x76F60000 - 0x76F8C000         C:\WINDOWS\system32\WLDAP32.dll^M
0x73760000 - 0x737A5000         C:\WINDOWS\System32\ddraw.dll^M
0x73BC0000 - 0x73BC6000         C:\WINDOWS\System32\DCIMAN32.dll^M
0x11C00000 - 0x11C3D000         C:\Program Files\Network Associates\VirusScan\Wb
^M
hook32.dll^M
0x77C00000 - 0x77C07000         C:\WINDOWS\system32\VERSION.dll^M
0x10000000 - 0x10009000         C:\Program Files\Network Associates\VirusScan\Re
^M
s09\WbhkRes.dll^M
0x76FC0000 - 0x76FC5000         C:\WINDOWS\System32\rasadhlp.dll^M
0x76C90000 - 0x76CB2000         C:\WINDOWS\system32\imagehlp.dll^M
0x6D510000 - 0x6D58C000         C:\WINDOWS\system32\DBGHELP.dll^M
0x76BF0000 - 0x76BFB000         C:\WINDOWS\System32\PSAPI.DLL^M
^M
Local Time = Tue Apr 23 10:51:38 2002^M
Elapsed Time = 1206^M
#^M
# The exception above was detected in native code outside the VM^M
#^M
# Java VM: Java HotSpot(TM) Client VM (1.3.1_04-ea-b01 mixed mode)^M
#^M
# An error report file has been saved as hs_err_pid632.log.^M
					 ^^^^^^^^^^^^^^^^^^ contains same info as above....
# Please refer to the file for further information.^M
#^M
====================================================================

Output from Linux 7.2 :

Exception in thread "AWT-Motif" Exception in thread "AWT-Motif" java.lang.OutOfM
emoryError
        <<no stack trace available>>
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
        <<no stack trace available>>
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
        <<no stack trace available>>
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
        <<no stack trace available>>
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError
Exception in thread "AWT-Motif" java.lang.OutOfMemoryError

After contact with engineer I increased the map heap to 128mb.
I've run the JDK with the -verbose:gc flag as well.

I let the JCK harness run overnight and the average heap size after a full GC cycle was 80-90 MB. No errors reported. This was with 131_04. One odd thing I noticed was that one full GC cycle took 287 seconds to complete. This was a rare event though (length of time) The particular GC reduced the heap from 128MB to 86 MB. The machine is a 1.8 GHz Pentium IV with 256MB of RAM running.

I've tried the same setup today with 131_fcs and the results look pretty similar. The heap size after 7hrs running time is pretty similar to 1.3.1_04 ~90MB. The problem will naturally be harder to reproduce with a bigger heapsize.

I'm going to compare both jdks with the 64MB heapsize again next week and see how the GC works with this. i.e. compare whether FCS can run the harness with 64MB heapsize without errors. 

The problem was initially produced by testing a 1.4.0 JDK with the JCK testharness with max heapsize of 64mb(which was run using 1.3.1_04) i.e.
JDKPATH="C:/jdk1.3.1_04b1"
$JDKPATH/bin/java -ms32m -mx64m javasoft.sqe.javatest.tool.Main  -startSlavePool 
-slavePoolPort 2901  $JTPFILE

I can install VNC on the desktop if engineers wish to log into the machine and modify settings.
  xxxxx@xxxxx   2002-06-28

I'm marking this bug "incomplete" until we (a) get instructions on 
how to reproduce it, and (b) get some guidance on whether it's a 
JVM problem or a JCK harness problem (-Xmx set too small).

  xxxxx@xxxxx   2002-06-27

----------------------------------------------------------------

> Subject: Re: Any status change on 4708390?
>    Date:  Fri, 9 Aug 2002 09:57:34 +0100 (BST)
>    From: Sean Coffey <  xxxxx@xxxxx  >
>      To:    xxxxx@xxxxx  
>      CC:   xxxxx@xxxxx  ,   xxxxx@xxxxx  
> 
> Hi Peter, 
> 
> The update that I gave was that increasing the memory heap works.
> 
> I also feel that whichever JDK you use to run the harness does not
> matter. It's whenever we're testing the 1.4 JDK, (~20,000 tests each
> executed in series with 1.4) we see the increased mem.  usage. Let's put
> this down to 3 possible reasons :
> 
> a) The fact that an extra ~8,000 (total ~20,000)test cases are used
> which increases the working memory heap to ~80-90 megs. ( running
> ~12,000 tests with 1.3 uses less than 64 MBs.)
> 
> b) The javatest harness storing too much information in RAM at any one
> time (test execution stats?)  and not freeing up memory correctly
> 
> c)  A leak somewhere in the jdk execution
> 
> I'm not sure where to go from here. The question is will customers see
> this increase in mem usage also when they run their apps with 1.4 ?
> 
> Note also that this was was for Merlin testing and not Hopper.  Please
> update the bug as you see necessary.
> 
> Regards, 
> Sean.

I'm willing to believe either (or both) a and b)  I'm not ruling out c, 
but we haven't seen any unexplained leaks in other tests, nor have 
we had complaints that users have seen increased footprints like 
this with JDK-1.4.x.

  xxxxx@xxxxx   2002-08-21

----------------------------------------------------------------