EVALUATION

Regarding the fix for JDK 7:

1. It cannot be directly fwd-ported since the code in question (the explicit check for the COMPIZ window manager) is not present in the XDecoratedPeer code. So basically it should be working normally.

2. However there's currently one obvious inconsistency in that code:

 long parent = XlibUtil.getParentWindow(window);
 Insets correctWM = (parent != -1) ? XWM.getWM().getInsets(this, window, parent) : null;

Note: the getParentWindow() returns 0 in case the window does not have a valid parent or the XQueryTree operation fails. However, the check for -1 was left from the previous version of the code (as it existed in JDK 6).

On the other hand 0 means None, which might be a perfectly valid value in some cases.

Besides all that, even if we change the check it won't change the logic since I think the parent can never be -1, so we always call the getWM().getInsets() now in JDK 7. And this, actually, is what the fix in JDK 6 does.
                                     

SUGGESTED FIX

--- old/src/solaris/classes/sun/awt/X11/XDecoratedPeer.java	2008-04-16 12:12:36.000000000 +0400
+++ new/src/solaris/classes/sun/awt/X11/XDecoratedPeer.java	2008-04-16 12:12:36.000000000 +0400
@@ -711,19 +711,15 @@
             if (!insets_corrected && getDecorations() != winAttr.AWT_DECOR_NONE) {
                 Insets correctWM;
 
-                if (runningWM == XWM.COMPIZ_WM) {
-                    correctWM = XWM.getWM().getInsets(this, window, XConstants.None);
-                } else {
-                    long parent = -1;
-                    XQueryTree qt = new XQueryTree(window);
-                    try {
-                        qt.execute();
-                        parent = qt.get_parent();
-                    } finally {
-                        qt.dispose();
-                    }
-                    correctWM = (parent != -1) ? XWM.getWM().getInsets(this, window, parent) : null;
+                long parent = -1;
+                XQueryTree qt = new XQueryTree(window);
+                try {
+                    qt.execute();
+                    parent = qt.get_parent();
+                } finally {
+                    qt.dispose();
                 }
+                correctWM = (parent != -1) ? XWM.getWM().getInsets(this, window, parent) : null;
 
                 if (insLog.isLoggable(Level.FINER)) {
                     if (correctWM != null) {
                                     

EVALUATION

it looks like the property is missing because of sime thread race between WM, us and
X-server.  It looks like the best way to fix this is do not read the property at random time, but listen for property change events fot it.
                                     

EVALUATION

possible (semioptimal) workaround of the problem is to pass real parent of the window into
XWM.getInsets() in handleConfigureEvent().  Right now for Compiz we pass None, since it should
be unneeded.  This way getInsets() will return (0, 0, 0, 0) insets in problematic case.
This may cause some problems, but not so critical.
                                     

EVALUATION

Finally I was able to find the test which reproduce the problem on my computer :)
After some tracing I've found that sometimes when we read extents (XGetWindowProperty())
in ConfigureNotify handler (XDecoratedPeer.handleConfigureNotify()) the window doesn't have
_NET_FRAME_EXTENTS property on it :(  Thus we calculate insets incorrectly.

I do not know for now why the property is missing sometimes.  Was unable to write native test
for this.
                                     

EVALUATION

After deeper investigation I've came to conclusion that the problem with moving dialog
is a Compiz problem.  So, I've developed small native test and filed bug against Compiz
(http://bugs.freedesktop.org/show_bug.cgi?id=13589)
                                     

EVALUATION

the cause of the problem with shifting of dialog is as follows:
we show dialog at some location (say 100x100),  later we receive ConfigureNotify with the
coordinates (100x100), but this coordinates are for shell, not for decorations, while
Dialog.getLocation() is supposed to return location of decorations, thus we recalculate
location (using insets).  As result we have adialog located (95, 76) (in my case instets 
are (24,5,5,5))
If now we dispose the dialog and show it again (w/o setting new location) the dialog will be
located at (90, 52), etc.

The difference with Metacity is that metacity places decorations at location we initially
specify for our window (shell).  I need to check which flags we use for this and see what we
need/can do to make Compiz behaves the same way.
                                     

EVALUATION

one more problem reported on the forum:

I'm not sure if this is related but, one oddity that I've noticed, which seems to happen consistently, is with the replace dialog opening in the wrong location:
1. Press CTRL-H to open the Replace dialog
2. Press ESC to dismiss it.
3. Repeat several times

Even if the replace dialog paints correctly, it will still crawl up the screen every time it opens until it gets to the top of the screen. Weirder still, this doesn't happen with CTRL-O. I thought it might be a problem with Netbeans but the problem does not happen under JDK5+MToolkit.

Need to investigate what is the cause of the problem.
                                     

EVALUATION

I have asked on the Swing&AWT forum (http://forums.java.net/jive/thread.jspa?threadID=33311) and it looks like the problem is
hardly reproducible: one needs to open/close dialog for some time.
In my environment I was able to reproduce the problem only twice (in two days).
Looks like I have to study the code and meditate :(