J2SE JRE 1.4.1_xx and 1.4.2, -server (C2) compiler only:

   Pure java application appears to be miscompiled by HotSpot C2.
   A method experiences an unexpected java.lang.NullPointerException as
   soon as it has been compiled.  Does not happen with -client (C1)
   compiler, does not happen with 1.4.0 and does not happen with
   -server -XX:-UseOnStackReplacement

   Customer Situation:
   Large portal application, using:
    TomCat4.0.3_LE-jdk14
    xerces 1.4.4
    castor 0.9.4
   Production environment still running on 1.4.0 and not affected by
   the problem.  Affected:  testbed environment for evaluating 1.4.1_01.
   As long as the bug exists, it means customer cannot migrate to 1.4.1.
   Problem is only reproduceable at the customers site, so we do not have 
   a testcase.

   Problem has first manifested in the following output:

   running with -XX:+PrintCompile we get the following output:

   ===8<
 ---
   73% !    org.exolab.castor.xml.Validator::validate @ 123  (278 bytes)
   442       org. customer .xerces.utils.StringPool::addSymbol (334 bytes)
   443  !    org. customer .catalina.util.RequestUtil::URLDecode (123 bytes)
   444       java.lang.ref.WeakReference:: (6 bytes)
   445  !    org. customer .catalina.core.ApplicationHttpRequest::setRequest (162 bytes)
   446       java.util.Hashtable$Enumerator::hasMoreElements (53 bytes)
   447       org. customer .oro.text.regex.Perl5Matcher::__match (2671 bytes)
   [29.10. customer  15:17:00] [Thread-16] [INFO ] [rontend.actions.IBBAction]
   [BF1F82F6E575A0786BCB114E86689F31] [Set referer to: /application_ibb_viewturnover_struts]
   448       java.lang.Class$1:: (15 bytes)
    74% !    org.exolab.castor.xml.UnmarshalHandler::processAttributes @ 320  (832 bytes)
   449       org.exolab.castor.xml.FieldValidator::validate (601 bytes)
   450  !    org.exolab.castor.mapping.loader.MappingLoader::createFieldDesc (1777 bytes)
   [29.10. customer  15:17:04] [Thread-17] [INFO ] [rontend.actions.IBBAction]
   [BF1F82F6E575A0786BCB114E86689F31] [Set referer to:
   /application_ibb_view_periodicmoneytransfer_struts]
   [29.10. customer  15:17:04] [Thread-17] [INFO ] [k.core.mock.MarshalFacade]
   [BF1F82F6E575A0786BCB114E86689F31] [before loadMapping
     xxxxx@xxxxx  ]
   [29.10. customer  15:17:05] [Thread-17] [ERROR] [k.core.mock.MarshalFacade]
   [BF1F82F6E575A0786BCB114E86689F31] [MappingException]
   java.lang.NullPointerException
           at org.exolab.castor.xml.Validator.validate(Validator.java:128)
           at org.exolab.castor.xml.FieldValidator.validate(FieldValidator.java:217)
           at org.exolab.castor.xml.Validator.validate(Validator.java:133)
           at org.exolab.castor.xml.FieldValidator.validate(FieldValidator.java:217)
           at org.exolab.castor.xml.Validator.validate(Validator.java:133)
           at org.exolab.castor.xml.UnmarshalHandler.endElement(UnmarshalHandler.java:675)
           at org. customer .xerces.parsers.SAXParser.endElement(SAXParser.java:1392)
           at org. customer .xerces.validators.common.XMLValidator.callEndElement(XMLValidator.java:1550)
           at org. customer .xerces.framework.XMLDocumentScanner$ContentDispatcher.dispatch(XMLDocumentScanner.java:1204)
           at org. customer .xerces.framework.XMLDocumentScanner.parseSome(XMLDocumentScanner.java:381)
           at org. customer .xerces.framework.XMLParser.parse(XMLParser.java:1098)
           at org.exolab.castor.xml.Unmarshaller.unmarshal(Unmarshaller.java:530)
           at org.exolab.castor.mapping.Mapping.loadMappingInternal(Mapping.java:507)
           at org.exolab.castor.mapping.Mapping.loadMapping(Mapping.java:433)
           at de.advancebank.core.mock.MarshalFacade.mapping(MarshalFacade.java:138)
   [...]
   --->
 8===

   When customer adds a .hotspot_compiler file prescribing
    exclude org/exolab/castor/xml/Validator validate
   the C2 compiler picks it up:
   ### Excluding compile:  org.exolab.castor.xml.Validator::validate
   and the problem no longer occurs, suggesting that the miscompile
   affects this method itself, and not one of its callers in the
   above stack.
   

   Now we still get Null Pointer Exceptions. But now in Unmarshaller.unmarshal.
   We have a lot of traces. See comments where. 

   traces look like this:

DEOPT UNPACKING thread 0x11ca988 vframeArray 0x12c9100
     {method} 'scanMatchingName' '( customer )I' in 'org/ customer /xerces/readers/UTF8Reader' - aastore @ bci 1083 sp = 0xe1
d7f5f8
!! !! SAXException in Unmarshaller.unmarshal
java.lang.NullPointerException
        at org. customer .xerces.framework.XMLParser.parse(XMLParser.java:1111)
        at org.exolab.castor.xml.Unmarshaller.unmarshal(Unmarshaller.java:533)
        at org.exolab.castor.mapping.Mapping.loadMappingInternal(Mapping.java:507)
        at org.exolab.castor.mapping.Mapping.loadMapping(Mapping.java:433)
        at de.advancebank.core.mock.MarshalFacade.mapping(MarshalFacade.java:139)
        at de.advancebank.core.mock.MarshalFacade.doUnmarshal(MarshalFacade.java:194)
        at de.advancebank.core.mock.MarshalFacade.unmarshal(MarshalFacade.java:172)
        at de.advancebank.business.services.banking.client.DemoBankingDelegate.getAssetsStatusInternal(Unknown Sou
rce)
        at de.advancebank.business.services.banking.client.BankingDelegate.getAssetsStatus(Unknown Source)
        at de.advancebank.portal.application.ibb.frontend.actions.StatusOverviewAction.doPerform(Unknown Source)
        at de.advancebank.portal.application.ibb.frontend.actions.IBBAction.doPerformAction(Unknown Source)
        at de.advancebank.portal.framework.webcomponents.frontend.ComponentAction.doPerform(Unknown Source)
        at de.advancebank.portal.framework.session.frontend.struts.PortalAction.perform(Unknown Source)
        at org. customer .struts.action.ActionServlet.processActionPerform(ActionServlet.java:1787)
        at org. customer .struts.action.ActionServlet.process(ActionServlet.java:1586)
        at org. customer .struts.action.ActionServlet.doGet(ActionServlet.java:492)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
        at de.advancebank.portal.framework.session.frontend.PortalKeeper.service(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at org. customer .catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
        at org. customer .catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
        at org. customer .catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:243)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)
        at org. customer .catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)
        at org. customer .catalina.core.ContainerBase.invoke(ContainerBase.java:943)
        at org. customer .catalina.core.StandardContextValve.invoke(StandardContextValve.java:190)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)
        at org. customer .catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:475)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)
        at org. customer .catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)
        at org. customer .catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)
        at org. customer .catalina.core.ContainerBase.invoke(ContainerBase.java:943)
        at org. customer .catalina.core.StandardContext.invoke(StandardContext.java:2343)
        at org. customer .catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)
        at org. customer .catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)
        at org. customer .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:170)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:564)
        at org. customer .catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)
        at org. customer .catalina.core.ContainerBase.invoke(ContainerBase.java:943)
        at org. customer .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
        at org. customer .catalina.core.StandardPipeline.invokeNext(StandardPipeline.java:566)
        at org. customer .catalina.core.StandardPipeline.invoke(StandardPipeline.java:472)
        at org. customer .catalina.core.ContainerBase.invoke(ContainerBase.java:943)
        at org. customer .catalina.connector.warp.WarpRequestHandler.handle(WarpRequestHandler.java:217)
        at org. customer .catalina.connector.warp.WarpConnection.run(WarpConnection.java:194)
        at java.lang.Thread.run(Thread.java:536)
Nested exception is:
java.lang.NullPointerException
[20.01.03 16:27:13] [Thread-23] [ERROR] [k.core.mock.MarshalFacade] [5B0F1791F680F15C0EE15DFC9645E69E] [!!!]
[20.01.03 16:27:13] [Thread-23] [ERROR] [k.core.mock.MarshalFacade] [5B0F1791F680F15C0EE15DFC9645E69E] [java.lang.
NullPointerException
]
[20.01.03 16:27:13] [Thread-23] [ERROR] [k.core.mock.MarshalFacade] [5B0F1791F680F15C0EE15DFC9645E69E] [MappingExc
eption]
java.lang.NullPointerException
[20.01.03 16:27:13] [Thread-23] [FATAL] [rontend.actions.IBBAction] [5B0F1791F680F15C0EE15DFC9645E69E] [exception 
caught:]
de.advancebank.core.error.InternalException: Nested error: java.lang.NullPointerException
        at de.advancebank.core.mock.MarshalFacade.doUnmarshal(MarshalFacade.java:217)
        at de.advancebank.core.mock.MarshalFacade.unmarshal(MarshalFacade.java:172)
        at de.advancebank.business.services.banking.client.DemoBankingDelegate.getAssetsStatusInternal(Unknown Sou
rce)

-client

-server -XX:-UseOnStackReplacement

exclude org/exolab/castor/xml/Validator validate

  xxxxx@xxxxx   2003-01-30

Possible duplicate of 4761344.
----- -----

Here is evaluation information copied from 4761344:

Attached file BadNullAssert.java shows a test case for a
possible root cause of this bug.

The problem is subtle: An OSR method is compiled using a model
of typestates which are derived from flow analysis over the
bytecodes.  This flow analysis is necessarily incomplete if the
interpreter has not yet exercised all paths in the code, leaving
some classes still unloaded.  In effect the CFG for the method
grows over time, as the interpreter exercises new paths and
loads new classes.  Exercising a new path must invalidate
pre-existing OSR methods, since they may have compiled into them
assumptions about typestates that must change as new CFG edges
come into being.

The problem can manifest (in the test case above, for example)
when an unloaded class is the subject of a checkcast operation.
The flow analysis that precedes an OSR compilation will simply
assert that any value casted to an unloaded class must be null,
and the compilation proceeds on that assumption.  Later on, if
the class loads, the interpreter can checkast non-null values
successfully, leading to typestates that were previously proved
impossible.  The OSR method can be invoked on a typestate
created by the interpreter containing non-null values, and yet
the OSR code will assume that they are null, based on the
out-of-date type analysis.

Solution is to have OSR methods register dependencies on
unloaded classes, and get flushed if the classes ever load.
This is not an issue with regular (non-OSR) methods, since a
normal method creates all its own typestates, from the method's
entry point forward.  There is no way for the interpreter dump
surprising typestates into a non-OSR method.

A binary which fixes this bug is here:
 /net/lukasiewicz/export/lukasiewicz1/4761344

There are product, optimized, and fastdebug flavors.
It is based on the current 1.4.2 baseline.

To gather more information, testing should use the following
extra flags:
  -XX:+UnlockDiagnosticVMOptions
  -XX:+LogCompilation
  -XX:LogFile=hotspot.log

The file "hotspot.log" will be written with diagnostic
information.  The name can be adjusted to any pathname.

(  xxxxx@xxxxx   2003-01-17)

If convenient, first try running in product mode without logging,
and see if this affects the bug.

In any case, also run the with the logging options turned on,
so I can analyze the log.  If you "kill -9" the application when
logging is turned on, the log will be truncated and incomplete.
Use Control-C, or "kill -HUP" or "kill -TERM", to give the VM
a chance to assemble the log on exit.

I am particularly interested in inspecting the compilation log
for the offending method Validator.validate.  In order to pin
down the offending bytecode, I would like a copy of Validator.class
and also Validator.java, if possible.

Based on the stack trace in the bug report (which reports line
128 as the location of the NPE), I can try to identify the
bad bytecode, and inspect its compilation in the log.

  xxxxx@xxxxx   2003-02-05