Productive WLS application server instances keep crashing every few
hours.  At least four events today.  Had been running fine for months
before this, and customer claimed they hadn't changed anything  (well,
maybe not in the setup, but as evidenced by an observation from BEA
support in the old CR#4736567, infinite recursions in WLS can be triggered by a self-referential .jsp's!

The point being to give the customer some indication of where in their code (or possibly in BEA's) they should look for the cause of the excessive recursion.

The customer has already rolled back their latest application software
change to the previous version, and has not seen another stack overflow
since, but has not had any significant client load on that WLS instance
either during the day, so this alone does not imply much.  I expect,
however, that the cause (the deep recursion) is absent in this previous
app software version, and thus a debug JVM with this older app software
version running on top would not gather any useful data, and would just
leave both us and the customer in doubt about the definitive cause for
longer.  Customer's primary short-term goal is to understand the failure
mechanism before Monday morning  (preferably tonight)  so that they can
continue production next week without having to worry about crashes every
couple of hours.

robert-martin.silo-  xxxxx@xxxxx   2005-2-05 20:18:17 GMT

N/A

Just going through some of the old bugs: this does sound like
problems with self-referential JSPs, and like the problem where
the JSP error page contained an error: I remember these demonstrated
the problem that 1.3.1 would crash rather then give the thread a
StackOverflowException, in older update releases.

Core files have been cleared away I think, so closing this as one of
those problems may be the only thing to do unless there's an ongoing
case with the customer.
Posted Date : 2006-05-11 11:53:05.0

I still have that (huge) core file, but I won't bother to attach it.
It did overflow a stack.

Citing my own old radiance case solution note:

==8<--
Further coredump post-mortem analysis identified the java method which was
being called recursively  (nested more than 2000 levels deep).  The recursion
was data-driven  (not a spontaneous infinite one),  roughly speaking it was
descending an XML document-object model of exorbitant depth.

Customer derived an application-side workaround for this issue, and is now
working on a modified implementation.  (I proposed either capping the model
depth in advance and adjusting the -Xss parameter to match the maximum depth,
or rewriting the method to work iteratively instead of recursively, which
could potentially scale to far larger depths yet.  Of course it would not
help much to go over the top with -Xss since all the thread stacks must also
somehow fit into the address space, and there were already about 400 threads
in the core.)

Side point:  1.3.1_12 or later would probably have thrown a clean
java.lang.StackOverflowError.  The unexpected SIGSEGV in the core showed
evidence of bug 4402735, a secondary issue occurring while we are already
in the middle of stack-overflow fatal-error handling;  this bug was fixed
in 1.3.1_12 and later.  But this was of little consequence - the error was
fatal in any case, and the process would have exited  (without a coredump)
anyway.
-->8==

This CR should thus probably be closed as a duplicate of 4402735.
Posted Date : 2006-05-11 12:59:48.0