EVALUATION

So far I got zero success with the proposed solution. However, there's also another issue (7174713) which seems to be realted to a memory leak caused by 2D code that operates with CALayers. These two issues may be related.
                                     

EVALUATION

Certainly, some improvements to the memory management code should be done. E.g. the CPlatformWindow is a CFRetainedResource. However, the CFRetain is currently called for the AWTWindow instance, whereas at the time of disposing a window, the corresponding CFRelease will be invoked for the NSWindow object. Also, the delegate object must be released manually (note that NSWindow setDelegate does NOT retain the delegate).

After making these changes I ensure that an instance of the AWTWindow class is in fact released and deallocated when Java calls removeNotify()/dispose() for a frame. And thus the javaPlatformWindow (which is a weak reference, but still) gets released as well. Also, the nsWindow property gets nil'ed, and hence doesn't hold the NSWindow object anymore.

However, the NSWindow object is still retained somewhere. I presume that 2D code holds a reference to the window. This code may also be responsible for creating a global ref to its Java Window object which prevents windows from being disposed completely. We should investigate this under 7174713.
                                     

SUGGESTED FIX

--- old/src/macosx/classes/sun/lwawt/macosx/CPlatformWindow.java	2012-06-22 19:45:41.000000000 +0400
+++ new/src/macosx/classes/sun/lwawt/macosx/CPlatformWindow.java	2012-06-22 19:45:41.000000000 +0400
@@ -63,6 +63,7 @@
     private static native void nativeSetNSWindowSecurityWarningPositioning(long nsWindowPtr, double x, double y, float biasX, float biasY);
     private static native void nativeSetEnabled(long nsWindowPtr, boolean isEnabled);
     private static native void nativeSynthesizeMouseEnteredExitedEvents(long nsWindowPtr);
+    private static native void nativeDispose(long nsWindowPtr);
 
     private static native int nativeGetNSWindowDisplayID_AppKitThread(long nsWindowPtr);
 
@@ -423,6 +424,7 @@
         EventQueue.invokeLater(new Runnable() {
             public void run() {
                 contentView.dispose();
+                nativeDispose(getNSWindowPtr());
                 CPlatformWindow.super.dispose();
             }
         });
--- old/src/macosx/native/sun/awt/AWTWindow.m	2012-06-22 19:45:42.000000000 +0400
+++ new/src/macosx/native/sun/awt/AWTWindow.m	2012-06-22 19:45:42.000000000 +0400
@@ -228,6 +228,7 @@
     }
 
     if (self.nsWindow == nil) return nil; // no hope either
+    [self.nsWindow release]; // the property retains the object already
 
     self.isEnabled = YES;
     self.javaPlatformWindow = platformWindow;
@@ -677,9 +678,9 @@
                                                   styleBits:styleBits
                                                   frameRect:frameRect
                                                 contentView:contentView];
+        // the window is released is CPlatformWindow.nativeDispose()
 
-        if (window) CFRetain(window);
-        [window release]; // GC
+        if (window) CFRetain(window.nsWindow);
     }];
 
 JNF_COCOA_EXIT(env);
@@ -1158,5 +1159,26 @@
     }];
 
 JNF_COCOA_EXIT(env);
+}
+
+JNIEXPORT void JNICALL Java_sun_lwawt_macosx_CPlatformWindow_nativeDispose
+(JNIEnv *env, jclass clazz, jlong windowPtr)
+{
+JNF_COCOA_ENTER(env);
+
+    NSWindow *nsWindow = OBJC(windowPtr);
+    [JNFRunLoop performOnMainThreadWaiting:NO withBlock:^(){
+        AWTWindow *window = (AWTWindow*)[nsWindow delegate];
+
+        // AWTWindow holds a reference to the NSWindow in its nsWindow
+        // property. Unsetting the delegate allows it to be deallocated
+        // which releases the reference. This, in turn, allows the window
+        // itself be deallocated.
+        [nsWindow setDelegate: nil];
+
+        [window release];
+    }];
+
+JNF_COCOA_EXIT(env);
 }
                                     

EVALUATION

Actually, only one more call to release is missing: we allocate a new NSWindow object and assign it to a property. We must release it manually since the property retains it automatically. After this change the PopupFactory test passes.
                                     

EVALUATION

This is a regression caused by 7149062.

Looks like the cause is that the nsWindow property is nil'ed in AWTWindow.dealloc. But dealloc is never called because the nsWindow holds a reference to the AWTWindow object (because the AWTWindow is set as a delegate for the nsWindow).