SUGGESTED FIX

The fix is to keep in AbstractPlugin class a variable which stores how many JavaScript calls 
currently are on the stack. 
If JVM on browser side receives a message to push a modality it checks if Java Scripts calls are alredy on the
stack. If so then a message to push a modality is being ignored (see ModalitySupport.java).
                                     

EVALUATION

More deatils from Hao:
"In the new plugin, when a modal dialog is showing, awt notify plugin by calling registered modality listener.  Plugin send a message from attached jvm to the browser side vm to block the browser main thread. While blocking the main thread, we still pump windows messages so a javascript invocation is possible.

When java invokes javascript, we do similar thing. A message sent to the browser main thread to invoke javascript. When the alert box is poped up, windows is pumping messages so a modality block can still happen on the browser main thread.

The testcase does modal dialog and javascript calls from two threads. They race with each other. On the browser main thread, the two can happen in either order. If javascript invocatio is the first, then the modality code will be on top it. Javascript call can not return unless the modality is popped (i.e. modal dialog is clicked).

I guess the customer want the javascript call can return first. To do this, they must ensure the javascript call happens after the modality push. There is no APIs to tell if the modality has pushed on the browser main thread. The closest thing is to check if a dialog is showing or not. When Dialog.setVisible() is called, awt calls plugin to push the modality. It takes time for plugin to send a message to browser side. If you start javascript immediately after the dialog is showing, they are still racing with each other. So let the javascript thead sleep 200 ms is a good heuristic."
                                     

WORK AROUND

.
                                     

EVALUATION

modality support block the javascript call because they all run on the browser main thread. If you click away the alert js dialog, js call can not return because there is modality calls on the stack.

You can change the code to make the modality dialog show up first. Then the js call will be on the top of the stack. Js call will return after the alert dialog is clicked. Then the modal dialog can be hidden.

In the attached TestApplet code, you can wait until the modal dialog is visible in the run() method before JSObject.getWindow() to call javascript. Then the js call will be pushed on top of the modality support on the main thread.