EVALUATION

From Jeff Nisewanger:

    The defaultDeny flag is set only for SocketPermission instances
created by the pre-FireFox 3 plugin for the purpose of
creating a ProtectionDomain representing the codebase JavaScript that
is directly invoking bootstrap loader classes (rather than invoking a
method from an associated applet). In that situation, we are relying on
the browser's DNS resolution of the script codebase to match the
plugin's independent DNS resolution of the same codebase host. Because,
the DNS resolutions and pinning databases are seperate the plugin cannot
safely rely that the browser downloaded the script from the same IP
address that the plugin would grant a SocketPermission to for the
ProtectionDomain representing the script codebase during the
JavaScript->Java invocation. To mitigate this, we set the defaultDeny
flag in the SocketPermission to require enhanced assurance that the
plugin's DNS resolution of the script codebase host looks legitimate
via a reverse DNS resolution check. If, instead, the script invokes
on a non-bootstrap class that is part of an associated applet then we
don't set the defaultDeny flag and just rely on the normal
SecurityManager check which will require both the script and applet
on the call stack to have SocketPermission grants that will approve
the network attempt and since the plugin/jvm downloads the applet code
it can use a consistent DNS pinning database for the applet codebase.
Before this fix, I was ham-handedly additionally setting the script
codebase into the "proxied host list" which persistently enforced the
enhanced DNS resolution checks in unnecessary situations.
                                     

SUGGESTED FIX

http://sa.sfbay.sun.com/projects/deployment_data/6u10/6669818.3
                                     

EVALUATION

In the fix for 6569621, the Socket class has been changed so that the checkConnect is always using the host address instead of sometimes using the host name as in prior to the fix.
Since this problem is affecting FF2.x with the old plugin only. 
A potential fix is to try to use the ip address when creating a SocketPermission in SecureInvocation which is only used in the old plugin on FF2.x.
                                     

EVALUATION

The previous evaluation wasn't entirely correct since I was looking at an outdated webrev for 6569621.
The cause of this problem is the following call in SecureInvocation.getDefaultProtectionDomain():
sun.net.www.URLConnection.setProxiedHost(host);

More investigation is needed since removing the above call may re-introduce the problem in 6569621.
                                     

EVALUATION

This problem is a regression caused by a security fix in 6u3 build 2 (CR 6569621) to prevent malicious exploitation of the fact that browser and plugin allowed communication via liveconnect even when DNS rebinding IP to a different hostname.

From the testcase: http://oueb.org/AppletFireFoxLiveconnectBug.html provided by Cyrille,
When clicking on "do Javascript call", there's a repinning hostname happened during the "JavaScript -> Java -> call to open connection with originating host" chain.
oueb.org IP is 213.186.33.4, but the reversed hostname for that IP from Sun DNS is 240plan.ovh.net.
The call from JS -> J is made with codebase oueb.org, but the Java open connection call is made with 240plan.ovh.net, which is different from codebase oueb.org and therefore the connection is prevented.

Cyrille also posted the same testcase on digimedia.fr. since digimedia.fr's IP is 91.121.68.177 and its reversed DNS hostname is also digimedia.fr, and therefore no issue there.

While 6569621 is a security fix, it may be too abrasive, and the fact that testcase from oueb.org works fine with IE shows inconsistency in behavior.
However, since 6569621 is a security fix, I will need to talk to Jeff to make sure whatever solution we come up with won't create another security hole.