This issue is related to 7144542. Please note that 7144542 is going to be addressed in 7u6 by loading another dynamic library for AWT (headless instead of headful) which will eliminate the crash in 7u6. However, this crash shouldn't be happening even with the wrong library loaded.
NOTE: 'java' below means a Java binary w/o a fix for 7144542, e.g. 7u4 GA or early builds of 7u6. This problem is not reproducible with JDK 8 since it implicitly contains a fix for 7144542.
NOTE2: early builds of 7u4/7u-osx (as of about Dec 2011) did not experience the issue. More over, even old Apple JDK 1.6 didn't experience it. However, since about Jan or Feb 2012 it has become reproducible with both 7u4 AND Apple JDK.
Steps to reproduce:
1. Build Java FX with a j2d-pipeline exit workaround removed:
$ hg clone http://jfxsrc.us.oracle.com/javafx/2.2/scrum/graphics/jfx fx
$ cd fx
$ hg clone http://jfxsrc.us.oracle.com/javafx/2.2/scrum/graphics/rt-closed
$ hg clone http://jfxsrc.us.oracle.com/javafx/2.2/scrum/graphics/rt
$ cd rt
$ patch -p 1 < /net/mandriva.ru.oracle.com/export/anthony/patch/fx-remove-mac-j2d-exit-workaround.patch
$ cd ..
(this usually takes ~10-15 minutes)
2. Build the HelloRectangle toy:
$ cd rt-closed/toys/HelloWorld/src
$ javac -classpath ../../../../artifacts/sdk/rt/lib/jfxrt.jar helloworld/HelloRectangle.java
3. Run it using the FX Prism J2D pipeline:
$ java -cp .:../../../../artifacts/sdk/rt/lib/jfxrt.jar -Dprism.order=j2d -Dprism.verbose=true helloworld.HelloRectangle
4. Click the red circle (Close button) in the title bar of the toy's window to exit the app, and observe the crash as described at 7144542:
Stack: [7fff65055000,7fff65855000], sp=7fff658523e0, free space=8180k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.dylib+0x27f3ae] ThreadStateTransition::trans_from_native(JavaThreadState)+0xa
V [libjvm.dylib+0x277a19] jni_CallStaticVoidMethodV+0x3d
C [JavaNativeFoundation+0x3c20] JNFCallStaticVoidMethod+0xae
C [liblwawt.dylib+0xde46] setBusy+0x79
C [CoreFoundation+0x63bd7] __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__+0x17
C [CoreFoundation+0x63b36] __CFRunLoopDoObservers+0x176
C [CoreFoundation+0x38dd6] __CFRunLoopRun+0x426
C [CoreFoundation+0x38676] CFRunLoopRunSpecific+0xe6
C [java+0x5cb4] CreateExecutionEnvironment+0x349
C [java+0x37b8] JLI_Launch+0x78d
C [java+0x7a30] main+0x6c
C [java+0x13f4] start+0x34