Please see issue : https://glassfish.dev.java.net/issues/show_bug.cgi?id=5250 for a detailed discussion
Posted Date : 2008-09-19 13:25:53.0

I am copying over the info from the jdk6.dev.java.net issue tracker.  The URL for 5250 is in the first Description entry.  The attachment mentioned has been added to this bug.

---begin---

Please see also Issue 5250 for Glassfish for more background information...

A JVM crash is noticed during a simple load test against Glassfish v2 ur b40
running any of the following JDK : jdk1.5.0_15, jdk1.6.0_06, jdk1.6.0_10rc
(build28).

By simply requesting the index.html on the SSL-enabled listener of the Glassfish
server, the process size (as reported by prstat) will increase steadily and
reach close to 4Gb (3.8) within 5 minutes.

hs_err_pidxxx.log and core file are available and can be generated as needed.

Most of them report one of the following 2 errors :

#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xff390858, pid=9667, tid=201
#
# Java VM: Java HotSpot(TM) Server VM (10.0-b22 mixed mode solaris-sparc)
# Problematic frame:
# C  [libc_psr.so.1+0x858]  memcpy+0x450
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#


or 


#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 140 bytes for CHeapObj-new. Out of swap
space?
#
#  Internal Error (allocation.inline.hpp:42), pid=9244, tid=118
#  Error: CHeapObj-new
#
# Java VM: Java HotSpot(TM) Server VM (10.0-b22 mixed mode solaris-sparc)
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

The crash though are always generated when the process is close to the 32bit
proc memory max size.

T2000 machines were used to run Glassfish and the load generator (Grinder). Both
machines are within the same subnet.

Running the same test against the HTTP listener (non-SSL) doesn't show any
leaks: the process size is very stable (no growth) and the process doesn't crash.

We've already involved the Glassfish team (see issue 5250) and after
investigation they recommended filing a bug against the JVM since all error
messages seem to point to a C heap corruption.

This is an urgent issue for the OpenSSO team.

Thanks,
N.

------- Additional comments from nphilipp Fri Aug 22 19:16:07 +0000 2008 -------

Created an attachment (id=6)
Tar file of all the hs_err_pid***.log files generated during testing

------- Additional comments from nphilipp Fri Aug 22 19:18:06 +0000 2008 -------

I also created a bug report earlier at bugreport.sun.com , just FYI.

Review ID: 1324946

---end---

I'm not sure what bug was filed at bugreport.sun.com.
Posted Date : 2008-10-20 23:21:33.0

N/A

Much of the IAIK code does not check for (malloc() == null) problems, which will lead to SIGSEGV's if memory is out.  This needs to be fixed, and will be a fair amount of work.

In the first example:  hs_err_pid6293.log, this clearly shows a null pointer being dereferenced following a call to wrapper_PKCS11_C_1EncryptUpdate, and a SIGSEGV being returned.

This is not the root cause of the memory leak, but definitely a problem which needs to be fixed.
Posted Date : 2008-10-21 17:58:49.0

The main problem was that the 128 threads were creating objects quickly, and only one Finalizer thread was being run.  As a result, there were a large number of native-heap backed Java objects waiting for finalization, and thus "leaking" memory. 
Also, the JSSE ciphers were never calling Cipher.doFinal(), and thus the Cipher objects had to cancel their PKCS11 backed cipher operations, which was also expensive and adding to the finalizer backlog.

We converted the P11Keys to use WeakReferences, plus added the doFinal in JSSE, and in the P11SecretKeyFactory to remove the final cancellation, and that took care of the finalization backlog.  Still showing a small amount of leaking, which is under investigation.  I put in some code to show PKCS11 native malloc/frees, and all are matching up.  Whereever this new leak is, it's not from the PKCS11 native calls.
Posted Date : 2008-11-26 22:17:18.0

We are facing exactly same crash with Java 1.5.0_15-b04 on Solaris (SunOS 5.10 Generic_127111-11 sun4v sparc SUNW,Sun-Fire-T1000). 

When can we expect this fix in Java 1.5?


Here is the snippet from hs_err_pid***.log file

==============================================================

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0xff390898, pid=503, tid=295
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_15-b04 mixed mode)
# Problematic frame:
# C  [libc_psr.so.1+0x898]  memcpy+0x450
#

---------------  T H R E A D  ---------------

Current thread (0x00e487a0):  JavaThread "jobs-236" daemon [_thread_in_vm, id=295]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000000

Registers:
 O0=0xe4c73200 O1=0x2ecdf0c4 O2=0x17030136 O3=0x575c829e
 O4=0x73aed053 O5=0x039ced4e O6=0x2ecddf20 O7=0x35e5563c
 G1=0x00000053 G2=0x00009400 G3=0x00009400 G4=0x00000000
 G5=0x00000000 G6=0x00000000 G7=0x2f9a2a00 Y=0x00000000
 PC=0xff390898 nPC=0xff39089c


Top of Stack: (sp=0x2ecddf20)
0x2ecddf20:   e4c73218 00000009 00000008 00000038
0x2ecddf30:   00004119 00000001 ff030f88 000083e4
0x2ecddf40:   00000000 e4c73220 00000018 00003680
0x2ecddf50:   ff030f84 00000001 2ecddf80 fe992f04
0x2ecddf60:   3c495a3e 2c1a8578 007a6c30 00000000
0x2ecddf70:   00584cc0 356c6bb0 356c6bb8 356c6bb4
0x2ecddf80:   e4c73218 e4c73224 fefdc000 0000369d
0x2ecddf90:   00004119 00000001 ff030f88 000083e4

Instructions: (pc=0xff390898)
0xff390888:   99 2b 10 12 83 33 50 13 98 13 00 01 c3 6a 20 40
0xff390898:   d6 f6 20 00 d8 f6 20 08 d4 5e 60 20 d6 5e 60 28

Stack: [0x2ecc0000,0x2ece0000),  sp=0x2ecddf20,  free space=119k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc_psr.so.1+0x898]  memcpy+0x450
V  [libjvm.so+0x192f0c]
C  [libj2pkcs11.so+0x5644]  Java_sun_security_pkcs11_wrapper_PKCS11_C_1DecryptUpdate+0x124
J  sun.security.pkcs11.wrapper.PKCS11.C_DecryptUpdate(JJ[BIIJ[BII)I
J  sun.security.pkcs11.P11Cipher.engineUpdate([BII[BI)I
J  com.sun.net.ssl.internal.ssl.CipherBox.decrypt([BII)I
J  com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Lcom/sun/net/ssl/internal/ssl/InputRecord;Z)V