I've found what looks like a long-standing memory leak involving the AWT focus subsystem.  I believe this is the root cause of bug 6462383 for 1.4.2 and 5.0, and possibly 6.0.

6462383 reports that even after a JFrame has been disposed and has no Java references, it is clearly still taking up space in the heap.  I was able to get YourKit to tell me that the frame was being kept alive via a JNI GlobalRef.  Some further debugging hinted that the most likely spot for the GlobalRef to be coming from was focus-related code.  In particular, awt_Component.cpp contains the following methods:

void *
AwtComponent::GetNativeFocusOwner() {
    JNIEnv *env = (JNIEnv *)JNU_GetEnv(jvm, JNI_VERSION_1_2);
    AwtComponent *comp =
        AwtComponent::GetComponent(AwtComponent::sm_focusOwner);
    return (comp != NULL) ? comp->GetTargetAsGlobalRef(env) : NULL;
}
void *
AwtComponent::GetNativeFocusedWindow() {
    JNIEnv *env = (JNIEnv *)JNU_GetEnv(jvm, JNI_VERSION_1_2);
    AwtComponent *comp =
        AwtComponent::GetComponent(AwtComponent::sm_focusedWindow);
    return (comp != NULL) ? comp->GetTargetAsGlobalRef(env) : NULL;
}

which both call GetTargetAsGlobalRef() in awt_Object.h:

    INLINE jobject GetTargetAsGlobalRef(JNIEnv *env) {
        jobject localRef = GetTarget(env);
        if (localRef == NULL) {
            return NULL;
        }

        jobject globalRef = env->NewGlobalRef(localRef);
        env->DeleteLocalRef(localRef);
        return globalRef;
    }

The functions above certainly show the possibilty for GlobalRefs to be created and then not be deleted.

I wrote an AWT-only test case which demonstrates this bug.  It should be run with -verbose:gc to show the memory leak when a normal Frame is created and disposed of.  Additionally, you can also see that an unfocusable Frame does not exhibit this leak - after a GC or two, the unfocusable Frame's memory drops out of the heap.

SimpleAWTTest.java:
-------------------
import java.awt.event.*;
import java.awt.*;

public class SimpleAWTTest extends Frame implements ActionListener {
    Button gcBtn;
    Button bigBtn;
    Button dspBtn;

    Button bigUnfBtn;
    Button dspUnfBtn;

    Frame bigFrame;
    Frame bigUnfFrame;

    public SimpleAWTTest() {
        super("SimpleAWTTest");
        Panel btnPnl = new Panel();
        btnPnl.setLayout(new FlowLayout());

        gcBtn = new Button("GC");
        gcBtn.addActionListener(this);
        btnPnl.add(gcBtn);

        bigBtn = new Button("Create Frame");
        bigBtn.addActionListener(this);
        btnPnl.add(bigBtn);

        dspBtn = new Button("Dispose Frame");
        dspBtn.addActionListener(this);
        btnPnl.add(dspBtn);

        bigUnfBtn = new Button("Create Unfocusable Frame");
        bigUnfBtn.addActionListener(this);
        btnPnl.add(bigUnfBtn);

        dspUnfBtn = new Button("Dispose Unfocusable Frame");
        dspUnfBtn.addActionListener(this);
        btnPnl.add(dspUnfBtn);

        add(btnPnl, BorderLayout.SOUTH);
        addWindowListener(new WL());
    }

    public void actionPerformed(ActionEvent e) {
        Object src = e.getSource();
        if (src == gcBtn) {
            System.gc();
        }
        else if (src == bigBtn) {
            bigFrame = new LeakFrame(true);
        }
        else if (src == dspBtn) {
            if (bigFrame != null) {
                bigFrame.dispose();
            }
            bigFrame = null;
        }
        else if (src == bigUnfBtn) {
            bigUnfFrame = new LeakFrame(false);
        }
        else if (src == dspUnfBtn) {
            if (bigUnfFrame != null) {
                bigUnfFrame.dispose();
            }
           bigUnfFrame = null;
        } 
    }

    static class LeakFrame extends Frame {
        byte[] bigLeak;

        public LeakFrame(boolean focusable) {
            super("Big Frame");
            bigLeak = new byte[1024 * 1024 * 24];
            setFocusableWindowState(focusable);
            if (focusable) {
                setBounds(0, 100, 100, 100);
            }
            else {
                setBounds(150, 100, 100, 100);
            }
            setVisible(true);
        }

        public void dispose() {
            System.out.println("dispose() called");
            super.dispose();
        }
    }
    
    static class WL implements WindowListener {
               public  void 	windowClosed(WindowEvent e) {
        }
        public  void 	windowActivated(WindowEvent e) {
        }
        public  void 	windowClosing(WindowEvent e){
            System.out.println("closing");
            Window src = ((Window)e.getSource());
            src.setVisible(false);
            src.dispose();
        }
        public  void 	windowDeactivated(WindowEvent e){}
        public  void 	windowDeiconified(WindowEvent e){}
        public  void 	windowIconified(WindowEvent e){}
        public  void 	windowOpened(WindowEvent e) {} 
    }

    public static void main(String[] args) {
        SimpleAWTTest f = new SimpleAWTTest();
        f.pack();
        f.setVisible(true);
    }
}

FWIW, this bug can be hard to pickup with a profiler.  It appears that there could be garbage collector action that (rarely) will pick up the wayward GlobalRef, though it's quite unreliable, and doesn't change the fact that we're creating GlobalRefs that don't get deleted.

Though long-standing, this has the potential to be a serious memory leak, depending on an application's architecture.  It should be fixed in 7.0 ASAP, and backported to all active update releases.
Posted Date : 2006-09-12 00:42:55.0

BTW, the test case is only equipped to handle 1 single focusable/non-focusable window at a time. :)
Posted Date : 2006-09-12 23:06:55.0

When running my test case, I've been able to workaround this bug by giving focus to some other application (I usually use a native terminal window), then back to the test case.

Along with returning a GlobalRef, AwtObject::GetTargetAsGlobalRef() could also store the global ref in an instance variable.  When the destructor is called, the GlobalRef could be deleted.
Posted Date : 2006-09-12 00:42:55.0

An interesting update: by giving focus to some other window (I used a native app), the leak will go away.  Do the following with the test case:

*Create Focusable Frame
*Dispose Focusable Frame

At this point, you can keep GCing and GCing and the leak will persist.  But as soon as you give focus to some other window, when you refocus the test window, GCing will clean up the heap.
Posted Date : 2006-09-12 23:54:07.0

We only call GetNativeFocusOwner() and GetNativeFocusedWindow() from KFM::GetNativeFocusState\90 which always deletes global ref as soon as it receives
(actually the only need of the globar ref here is to pass reference from one thread
to another.)
Thus I doubt that the cause of the leak we see is in that code.
Posted Date : 2006-09-27 11:03:36.0

it is realOppositeComponent and realOppositeWindow fields of DKFM who keeps strong references to disposed window/component.  It looks like that the easiest/best way to fix this is to convert these fileds into weak references.
Posted Date : 2006-10-13 10:19:13.0