EVALUATION

According to the Evaluation from the Comments section, it appears that FX/Glass have already detached the AppKit thread from the JVM, while AWT is unaware of this. It continues to call Java methods from the setBusy obersvers, and this leads to this crash.

Looks like FX has to be able to detect that the AWT is being running, and not detach from the AppKit thread in this case to make sure it is AWT who'e going to cleanup all things and finally terminate the process.
                                     

EVALUATION

I've added an AttachCurrentThreadAsDaemon() call to the setBusy() function to make sure the env isn't stale. Note that the Attach call never failed - i.e. it always returned a valid, non-null env.

Also, I got rid of all the JNF syntax sugar and used the real JNI calls (like (*env)->FindClass(env...), GetStaticMethodID, etc.). Also, I removed the 'static' caches and fetch all the values each time the setBusy() is executed. After I do this, I get a different stack trace for the crash:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00007fff8c42ece2 __pthread_kill + 10
1   libsystem_c.dylib             	0x00007fff912357d2 pthread_kill + 95
2   libsystem_c.dylib             	0x00007fff91226a7a abort + 143
3   libjvm.dylib                  	0x0000000101bb3397 os::abort(bool) + 25
4   libjvm.dylib                  	0x0000000101ca1632 VMError::report_and_die() + 2306
5   libjvm.dylib                  	0x0000000101bb4a82 JVM_handle_bsd_signal + 1064
6   libsystem_c.dylib             	0x00007fff91287cfa _sigtramp + 26
7   libjvm.dylib                  	0x0000000101aa3422 ThreadStateTransition::trans_from_native(JavaThreadState) + 10
8   libjvm.dylib                  	0x0000000101a927b6 jni_FindClass + 50
9   liblwawt.dylib                	0x000000010dec3e05 setBusy + 123

In this case the FindClass() call fails. All this indicates that the VM state is corrupt at this point.

Another interesting detail: after making a FindClass call I'm making a new global ref for the returned class reference. Nonetheless, the FindClass returns a different reference from time to time while the program is running (I've added an fprintf() in the setBusy() to see each and every call of this function). This seems strange as with a global ref in native, I would expect the FindClass to return the same reference each time I call it.
                                     

EVALUATION

The hs_err log reports "Current thread is native thread" which means that when the SEGV occurred ThreadLocalStorage::get_thread_slow() returned NULL. This would all seem to indicate that the current thread is calling into Java but is not attached to the VM. (Or else it's TLS storage has been corrupted).
                                     

EVALUATION

What has been tried to resolve the issue but didn't help:

1. Copied JVM 24b07 from JDK 8 to my build of 7u6.

2. Verified that in the setBusy() function (see src/macosx/native/sun/awt/awt.m in JDK) the env is always non-null and that the thread the function is executing on is always attached to the JVM.

Neither of the above helped resolve the issue.