EVALUATION

We will fix this particular problem by avoiding the actual setDisplayChange()
call when the given mode is the same as the current mode.  This avoids the OOME
so that this test and other apps that use only setFullScreenWindow() (and do
not change the display mode) will now work properly.  However, the underlying
cause of the OOME when calling setDisplayMode() repeatedly is still present
and will be handled under a separate bug (6459841).  (That issue is less
important because it only affects testcases that really stress the calling
of setDisplayMode() frequently, which is not a common scenario in the real world.)
                                     

EVALUATION

The test itself doesn't call setDisplayMode. So it appears that we
  call it during the setFullScreenWindow() call (as per your
  evaluation).

  Looks like we do it when restoring the original display mode.
  In this case the display mode doesn't change, so there is no need
  for calling setDisplayMode.

  We do the same thing on windows, except that in
  Win32GD.setDisplayMode we check if the passed display mode is the
  same as the current mode, and noop in this case.

  We don't do this on Unix, though (see X11GD.setDisplayMode()) - not
  sure why - so the display mode change event gets generated, and so
  on.

  Anyway, what I'm saying is that this test shouldn't really be
  causing all that carnage with display mode changes etc.

  Also, on my solaris system I can't reproduce the failure because the
  display mode changes aren't supported. 

Reassigning back to unix FS engineer (aka the victim).
                                     

EVALUATION

The OOME comes from the fact that the testcase creates a new Frame
(--> a new BufferStrategy --> a new VolatileImage --> a new "backup"
BufImageSurfaceData) for each iteration.  For each iteration we have:
  --> X11GD.setFullScreenWindow()
    --> X11GD.setDisplayMode()
      --> VSM.displayChanged()
        --> VSM.getBackupSurface()

Each "backup" surface created is about the size of the screen (in my case
that's 1600x1200) and is created in the Java heap, so if these are not
disposed quickly, they can easily exhaust the Java heap and cause the OOME.

I'm not sure what we can do to prevent this, but it's not a scenario that
most fullscreen apps are likely to run into.  Transferring back to the
test's owner while we give this some more thought.