Customer problem description:
------------------------------
When this server is hit with a load of HTTP requests VM crashes.
They are using jdk1.3.1 -server option and  get the following JVM crash.
>
>#
># HotSpot Virtual Machine Error, Internal Error
># Please report this error at
># http://java.sun.com/cgi-bin/bugreport.cgi
>#
># Error ID: 53414645504F494E540E43505002EA
>#
>abnormal program termination

We ran with the -XX+ShowMessageBoxOnError flag.  I got the stack trace from Visual C++ debugger. 
In a sample dry run, the java executable grows in sizeand then OutOfMemoryError appeared in the log just after verbose gc output.

Testcase:
We have things set up so that you should be able
to get access to the system running the tests.

In a browser, connect to the URL //205.180.14.95:5800
This will give you a VNC loggin screen.  The password
is "sunjava".  Once you log in, you will have an image
of the machines monitor.

One of the windows is running Buyer.  I believe we are
still running with MessageBox on, so you will be able
to get into Visual C++ from there.

You have to enable your proxy settings the following way:
remove your proxy configuration and add pluto.ebay as the socks 
host with port:1080

Please assign someone with  customer  skillset. As Ariba is known to be a trouble maker.

None

Based on email with Mir, I'm sending this back as 'incomplete'.
Mir was going to re-test with the 1.3.1_02 which might have fixed it,
or escalate with CTE if not.  If it escalates I'm sure it will come
back to me soon enough.  :-)

  xxxxx@xxxxx   2001-11-02

  xxxxx@xxxxx   2002-02-05

Analysed the memory dump file(drwtsn.dmp) sent by the customer using WinDbg. The dump file is available in the Attachments. Found out the offending method name and its class for which the bad oop was getting generated.

* The method name is found to be:
	java.lang.Object nextElement();

of class "ariba.common.core.ObjectArrayEnumerator".

The customer has been suggested the workaround to disable the compilation of the above method.


Analysis of crash dump(drwtsn.dmp)
----------------------------------

Class of the ofending method
----------------------------
codeblob* is (which is nmethod *) = 0x00fa3490

methodOop = *(0x00fa3490+0x2c) = 0x392905f0

Constant Pool = *(0x392905f0+0x08) = 0x39290370

pool-holder(klassOop) = *(39290370+0x14) = 0x39290698

Class name symbolOop = *(0x39290698+0x24) = 0x38fd7a48

Class name = (char*)(38fd7a48 + 0xe) = ariba/common/core/ObjectArrayEnumerator

------------------------------------------------------------------------------

Offending method name
---------------------
method name index is 24 bytes from the start of methodOop.
method signature index is 26 bytes from the start of methodOop.

name index =  37
signature index = 11

method signature symbolOop = *(constantpool + 24 + signatureindex*4) = 0x380e24d0

method signature = (char*)(0x380e24d0 + 0xe) = ()Ljava/lang/Object;

method name symbolOop = *(constantpool + 26 + nameindex*4) = 0x380f9a00

mathod name = (char*)(0x380f9a00 + 0xe)  = nextElement

-------------------------------------------------------------------------------

  xxxxx@xxxxx   2002-02-05

----------------------------------
Further analysis of the memory dump:

frame* (first parameter of OopMapSet::oops_do() = 0x3a4ffde8

_pc is at offset 4 from start of frame. = 0x00fa35d2 

From nmethod:  entry_point = 0x00fa3560   
	      _instructions_offset = 0x000000d0 = 208	
	      _oops_offset = 0x00000268 = 616	
	      _scopes_data_offset = 416
	      _scopes_pcs_offset = 506
	      _handler_table_offset = 616

instructions_begin = (nmethod*)+_instructions_offset = 0x00fa3560
_scopes_data_begin = (nmethod*)+_scopes_data_offset = 0x00fa3630
_scopes_data_end = (nmethod*)+_scopes_pcs_offset = 0x00fa3688
_scopes_pcs_begin = (nmethod*)+_scopes_pcs_offset = 0x00fa3688
_scopes_pcs_end = (nmethod*)+_handler_table_offset = 0x00fa36f8
oops_begin  = (nmethod*)+_oops_offset = 00fa36f8

pc offset = pc - instruction begin  = 0x72 = 114

Now, search for this pc offset in the pcDesc array starting from 
_scopes_pcs_begin.
The next entry is scopes_decode_offset = 0xffffffbd = (-67) negative sign means 
not at call(It is jmp)

scopes data buffer = _scopes_data_begin + scopes_decode_offset = 0x00fa35ed

From raw_int:

method oop index = 12
bci = 57

From method oop index, the methodoop of inlined method is:

methodOop -> oops_begin + (method oop index)*4
	  = 0x00fa36f8 + (12)*4 = 392905f0 which is same as the original 
methodOop.


Instruction at the _pc is:

	00fa35d2 0f8cbfffffff     jl      00fa3597

As 00fa3597 < 00fa35d2 , this is backward jump. 
So safepoint at backward jump contained a bad oop.
-------------------------------------------------------------- 
  xxxxx@xxxxx   2002-02-07