This crash occurred while doing a J2EE build with jdk1.3.1_03 on solaris sparc.

Date: Mon, 17 Jun 2002 16:09:20 -0700 (PDT)

Subject: HotSpot Crash



Does this look like a C1 crash to you guys?  It happens more frequently
on a big server (E250, E450).  Less frequently on Ultra 5:


Unexpected Signal : 11 occurred at PC=0xfb030734
Function name=compareTo (compiled Java code)
Library=(N/A)

Current Java thread:

Dynamic libraries:
0x10000 	
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/bin/../bin/sparc/nativ
e_threads/java
0xff350000 	/usr/lib/libthread.so.1
0xff390000 	/usr/lib/libdl.so.1
0xff200000 	/usr/lib/libc.so.1
0xff330000 	/usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
0xfe480000 	
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/client/l
ibjvm.so
0xff2d0000 	/usr/lib/libCrun.so.1
0xff1e0000 	/usr/lib/libsocket.so.1
0xff100000 	/usr/lib/libnsl.so.1
0xff0d0000 	/usr/lib/libm.so.1
0xff300000 	/usr/lib/libw.so.1
0xff0b0000 	/usr/lib/libmp.so.2
0xff080000 	
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/native_t
hreads/libhpi.so
0xff050000 	
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libverif
y.so
0xfe440000 	
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libjava.
so
0xff020000 	
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libzip.s
o
0xfe230000 	
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libioser
12.so

Local Time = Mon Jun 17 14:17:52 2002
Elapsed Time = 4
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002BD 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Client VM (1.3.1_03-b03 mixed mode)
#

You can create a .hotspot_compiler file in the directory where you run your program containing the line

exclude java/lang/String compareTo

This will keep us from compiling this method.  You'll get extra output as well though, looking kind of like this.

smite ~ % cat .hotspot_compiler
exclude java/lang/String compareTo
smite ~ % /export/ws/hopper/sparc/jdk1.4/bin/java stringcompare
CompilerOracle: exclude java/lang/String compareTo
### Excluding compile:  java.lang.String::compareTo

You may also be able to tweak your initial heap size using -Xms to perturb the location of the string which happens to end up at the end of the heap.
  xxxxx@xxxxx   2002-07-16

There's a load in the delay slot of the main compare loop that should be annulled.  This is very hard to reproduce since you have to compare a String whose char[] is the last object in the tenured generation.  The fix is simple but there isn't really a test case.
  xxxxx@xxxxx   2002-06-18

Date: Tue, 18 Jun 2002 09:59:05 -0700 (PDT)

Subject: Re: HotSpot Crash



I think I know what it is.  We emit a special implemenation of String.compareTo 
and whoever wrote it put a load in the delay slot of a branch without annulling 
it, so it loads one char past the end of the array.  The string being compared 
appears to be right at the end of a generation, so it's actually loading memory 
outside of any allocated space.  This used to be harmless but one of the changes 
we put back into 1.3.1_03 was to modify os::uncommit_memory to use PROT_NONE so 
that touching uncommitted memory would result in a crash and I think that's 
what's happening from looking at the core file.  I think you can make the 
problem disappear by picking some different heap parameters since that should 
perturb the allocation locations and put the string somewhere else.

We should probably fix this in 1.3.1_04 and in mantis since I think we just 
added to PROT_NONE change to mantis a couple weeks ago.  I've tried to write a 
test case for this but haven't had much luck, which is good I guess.  I'll file 
a bug for this.

tom

> Date: Mon, 17 Jun 2002 16:40:38 -0700 (PDT)
> From: Rajiv Mordani <  xxxxx@xxxxx  >
> X-Sender:   xxxxx@xxxxx  
> To: Thomas Rodriguez <  xxxxx@xxxxx  -mail1.SFBay.Sun.COM>
> cc:   xxxxx@xxxxx  -mail1.SFBay.Sun.COM,   xxxxx@xxxxx  ,   xxxxx@xxxxx  , 
  xxxxx@xxxxx  -mail1.SFBay.Sun.COM
> Subject: Re: HotSpot Crash
> 
> The core file is at /net/wampum/export/mode/wspack/core
> 
> 
> - Rajiv
> --
> :wq
> 
> On Mon, 17 Jun 2002, Thomas Rodriguez wrote:
> 
> > It's dying in compiled code so it's most likely a C1 bug.  It's rare for 
runtime 
> > bugs to cause crashes in compiled code.  Is there a test case or core file?
> > 
> > tom
> > 
> > > Date: Mon, 17 Jun 2002 16:09:20 -0700 (PDT)
> > > From: Janet Koenig <  xxxxx@xxxxx  >
> > > Subject: HotSpot Crash
> > > To:   xxxxx@xxxxx  -mail1.SFBay.Sun.COM,   xxxxx@xxxxx  -mail1.SFBay.Sun.COM, 
> >   xxxxx@xxxxx  
> > > Cc:   xxxxx@xxxxx  ,   xxxxx@xxxxx  -mail1.SFBay.Sun.COM
> > > 
> > > Does this look like a C1 crash to you guys?  It happens more frequently
> > > on a big server (E250, E450).  Less frequently on Ultra 5:
> > > 
> > > 
> > > Unexpected Signal : 11 occurred at PC=0xfb030734
> > > Function name=compareTo (compiled Java code)
> > > Library=(N/A)
> > > 
> > > Current Java thread:
> > > 
> > > Dynamic libraries:
> > > 0x10000 	
> > > 
> > 
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/bin/../bin/sparc/nativ
> > > e_threads/java
> > > 0xff350000 	/usr/lib/libthread.so.1
> > > 0xff390000 	/usr/lib/libdl.so.1
> > > 0xff200000 	/usr/lib/libc.so.1
> > > 0xff330000 	/usr/platform/SUNW,Ultra-250/lib/libc_psr.so.1
> > > 0xfe480000 	
> > > 
> > 
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/client/l
> > > ibjvm.so
> > > 0xff2d0000 	/usr/lib/libCrun.so.1
> > > 0xff1e0000 	/usr/lib/libsocket.so.1
> > > 0xff100000 	/usr/lib/libnsl.so.1
> > > 0xff0d0000 	/usr/lib/libm.so.1
> > > 0xff300000 	/usr/lib/libw.so.1
> > > 0xff0b0000 	/usr/lib/libmp.so.2
> > > 0xff080000 	
> > > 
> > 
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/native_t
> > > hreads/libhpi.so
> > > 0xff050000 	
> > > 
> > 
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libverif
> > > y.so
> > > 0xfe440000 	
> > > 
> > 
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libjava.
> > > so
> > > 0xff020000 	
> > > 
> > 
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libzip.s
> > > o
> > > 0xfe230000 	
> > > 
> > 
/net/koori.sfbay/a/v01/jdk/1.3.1_03/fcs/binaries/solsparc/jre/lib/sparc/libioser
> > > 12.so
> > > 
> > > Local Time = Mon Jun 17 14:17:52 2002
> > > Elapsed Time = 4
> > > #
> > > # HotSpot Virtual Machine Error : 11
> > > # Error ID : 4F530E43505002BD 01
> > > # Please report this error at
> > > # http://java.sun.com/cgi-bin/bugreport.cgi
> > > #
> > > # Java VM: Java HotSpot(TM) Client VM (1.3.1_03-b03 mixed mode)
> > > #
> > > 
> > 
> 


  xxxxx@xxxxx   2002-06-18

I realized that the other condition for seeing this bug is that the end of the tenured generation must also be exactly at the end of page, otherwise you won't see this failure.  So basically it requires comparing a string whose char array is the last object in tenured and tenured ends exactly on a page boundary. A fairly amazing confluence of events.

I also updated suggested fix to correspond to 1.3.1 sources instead of 1.4.1.
  xxxxx@xxxxx   2002-06-20

As someone pointed out it doesn't seem like an amazing confluence of events when it happens to you every time but it is.  It's just very deterministic as well so if you have a program which see it your are likely to see it in the future.
  xxxxx@xxxxx   2002-07-16

Yeah, right. "A fairly amazing confluence of events". So 
that's no more than maybe a zillion to one chance so it 
only happens about a hundred times a day. We're seeing this 
in production since recently switching to 1.3.1_03 and its 
bringing down machines.

This is the same bug as in report 4701709.

How about these for workarounds:

1. Use -Xms<max heap size> to allocate all the heap space 
in advance. 
2. Use "exclude String compareTo" in .hotspot_compiler to 
prevent compilation of the offending method. This has to be 
easier than excluding every method that uses "compareTo" as 
suggested in the other bug report.

Are either/both of these reasonable?

Guys I'm hiting this bug on :
Java VM: Java HotSpot(TM) Client VM (1.3.1_04-b02 mixed 
mode), So Imagine it hasen't been fixed in this compile, 
can anyone point out a 1.3.* distribution that would not 
have this bug, it's very disturbing to see thid error in my 
logs, especialy on such a trivial function ( comareTo() )

This bug is not closed. I'm hiting this bug in  Java
HotSpot(TM) Client VM (1.3.1_04-b02 mixed 
mode) too.

I'm encountering this problem running >> Java VM: Java 
HotSpot(TM) Client VM (1.3.1_04-b02 mixed mode) <<. So 
will increasing the heap sz (over default) get rid of this 
problem for definate? Or is it safer to omit the method when 
compiling ?

We run into the same problem on solaris with  Java
HotSpot(TM) Client VM (1.3.1_04-b02 mixed mode) today. Looks
like it's not fixed in the latest version.

About using "exclude String compareTo", is there a place to
put the .hotspot_comiler file  and it will be picked up no
matter which directory I start the server from?

Are you going to release 1.3.1_07 any time soon, or are we
going to have to replace Solaris as our server OS?

It is still open for 1.3.1_06-b01, anyway, it seems 
reproducible everytime, indexing about 130000 documents 
under jakarta lucene. We dont even have to use our own 
code, the indexing demo that comes with lucene fails 
everytime. However preventing java/lang/String compareTo 
from compiling works, but there might be a performance hit 
for something like indexing documents that might use this 
method quite heavily.

The bug is still here on 1.3.1_07, sol7 with dec 2002 
recommended patchset installed

I got the same roblem on Linux with:
java version "1.3.1"
Java(TM) 2 Runtime Environment, Standard Edition (build 
Blackdown-1.3.1-FCS)
Java HotSpot(TM) Client VM (build Blackdown-1.3.1-FCS, 
mixed mode)

A tool Source+JarExecutable which simulate the Problem can 
be found:
http://stud.fbi.fh-darmstadt.de/~doubrava/SFLists/sfclient.zip

Start the tool with: 
java -jar sflist.jar Ro.Do@gmx.de johan ./ 2

After some Scan Action the MiniApp will crash with the Error...

I see this error only on Linux OS on my Windows with same 
JVM Release the error didn't come ?!?!?

Thx....