EVALUATION

-Xdock:icon has nothing to do with the representation of minimized windows in the dock. The icon for a minimized window is specified with the Window.setIconImages() API.
                                     

EVALUATION

CPlatformWindow.getImageForTarget() must walk through images to choose the largest one.
                                     

SUGGESTED FIX

--- old/src/macosx/classes/sun/lwawt/macosx/CPlatformWindow.java	2012-03-08 20:14:57.000000000 +0400
+++ new/src/macosx/classes/sun/lwawt/macosx/CPlatformWindow.java	2012-03-08 20:14:56.000000000 +0400
@@ -725,9 +725,17 @@
             return null;
         }
 
-        // TODO: need a walk-through to find the best image.
-        // The best mean with higher resolution. Otherwise an icon looks bad.
-        final Image image = icons.get(0);
+        // Choose the best (largest) image
+        Image image = icons.get(0);
+        // Assume images are square, so check their widths only
+        int width = image.getWidth(null);
+        for (Image img : icons) {
+            final int w = img.getWidth(null);
+            if (w > width) {
+                image = img;
+                width = w;
+            }
+        }
         return CImage.getCreator().createFromImage(image);
     }
                                     

EVALUATION

The problem is reproducible. Note that even with Apple JDK the minimized icon is slightly worse in quality comparing to the running app icon. However, with Oracle JDK 7u4 the minimized icon looks very low-quality.