EVALUATION

After additional investigation (see Evaluation for 7166725), I think the best way to deal with this issue is to:

1. Revert the fix for 7144542, and
2. Not install the Busy observer in the headless mode.

Here's some background. An AWT app keeps running because the EDT is a non-daemon thread. The logic of the AWTAutoShutdown is based on the following:

1. While there are active peers, or
2. there are Java events in the EventQueue, or
3. there are native events on the toolkit thread (controlled by the Busy/Free notifications)

the EDT keeps the JVM from exiting. Once all three conditions are false, the AWTAutoShutdown will force the EDT to terminate. If an app doesn't have any more non-daemon threads it will terminate.

An app in the headless mode can't create peers. It is unlikely to use Java events either. But it may do so at any time it wishes. If the EDT doesn't exist, it will be created as soon as a user posts an event. Native events are unlikely to occur in the headless mode either because there's no native peers to generate them. Nonetheless, if native code calls performSelector (or otherwise generates an activity on the AppKit thread), this will work as expected since the AppKit thread is running in the headless mode anyway.

If we remove the Busy/Free observer in the headless mode on the Mac, this shouldn't really change anything since the native toolkit thread was 'free' almost all the time in this mode anyway. And even if an app posts Java events (which is unlikely, but still possible), AWT is ready to handle this situation by creating/destroying the EDT "on demand". Therefore, we don't see any problems with removing the observers.

I've verified that this change eliminates the crash when exiting an FX app running with the j2d pipeline on the Mac (i've modified the FX RT to remove the workaround that calls System.exit() to termiante an app in order to test this change). So with this fix we won't make thing worse than they were after fixing 7144542.

Also, reverting the fix for 7144542 enables the CToolkit in the headless mode, which makes the HeadlessPrintingTests pass, and thus resolves this bug (7174704).
                                     

EVALUATION

This regression is caused by the fix for CR 7144542 ([macosx] Crash in liblwawt.dylib setBusy() when exiting an FX app):
it prevents loading lwawt library in headless mode, that causes observed exception.

I am dispatching this bug to author of the fix 7144542  for further evaluation.
                                     

EVALUATION

When the headless mode is enabled, the XToolkit library is used as a headless AWT implementation on the Mac. However, the printing code still choosed the Cocoa-based printing implementation which relies on code present in the lwawt dynamic library.

There are several options to fix this:

1. Investigate if it's technically possible to use the XToolkit headless printing stack on the Mac. If it is possible, then we must use XToolkit printing support in the headless mode on the Mac.

2. Extract the printing native code to a separate library, and make the printing code on the Mac load this library.

3. The code from #2 could be compiled into the XToolkit headless native library.