SUGGESTED FIX

The behaviors change back for better backwards compatibilities.
                                     

WORK AROUND

As a workaround, I would suggest always convert the hostname into a resolved hostname before call the default checkClientTrusted in the customized trust manager,
_hostname_ = InetAddress.getByName(hostname).getHostName();
Or
_hostname_ = InetAddress.getByName(hostname).getCanonicalHostName();
                                     

PUBLIC COMMENTS

The HTTPS spec said, "Typically, the server has no external knowledge of what the client's identity ought to be and so checks (other than that the client has a certificate chain rooted in an appropriate CA) are not possible. If a server has such knowledge (typically from some source external to HTTP or TLS) it SHOULD check the identity as described above<server identity>." I think maybe the DNS could be considered as an external source to TLS.
                                     

EVALUATION

The application enable client identity checking during extending X509ExtendedTrustManager with a fixed identification algorithm, "https", and the bug fix of 6614957 changes the behaviors of how to get a ssl socket host name, cause known.
                                     

PUBLIC COMMENTS

Xuelei Fan wrote:

> Cause known, the application enable client identity checking during 
> extending X509ExtendedTrustManager with a fixed identification 
> algorithm, "https", and the bug fix of 6614957 changes the behaviors of 
> how to get a ssl socket host name.

Yes, the fix for 6614957 has clearly caused a regression.

> Yes, we need to find a general way for client identity. The HTTPS spec 
> said, "Typically, the server has no external knowledge of what the 
> client's identity ought to be and so checks (other than that the client 
> has a certificate chain rooted in an appropriate CA) are not possible. 
> If a server has such knowledge (typically from some source external to 
> HTTP or TLS) it SHOULD check the identity as described above<server 
> identity>." I think maybe the DNS could be considered as an external 
> source to TLS.

Yes, it may not be commonly-used, but it *is* supposed to work, and the spec says that if such a mechanism exists, it should be used.  That's the case here.

> I don't think I have enough time to fix it into 6u12 b01, I'd like have 
> a try.

Do you have a pointer to the build and release schedules?

> Because 6u10 is a major upgrade, I think you maybe need a workaround, I 
> would suggest always convert the hostname into a resolved hostname 
> before call the default checkClientTrusted in the customized trust manager,
> _hostname_ = InetAddress.getByName(hostname).getHostName();
> Or
> _hostname_ = InetAddress.getByName(hostname).getCanonicalHostName();

That should work, I'll check it out.

However I'm concerned by your assertion that the move from 1.6.0_07 to 1.6.0_10 "is a major upgrade".  According to http://java.sun.com/j2se/versioning_naming.html, the switch from _07 to _10 designates an "update release" which "contains customer focused bug fixes".  As far as I'm aware, that means that we shouldn't be introducing any major changes in existing functionality in such a release - something that has clearly happened in this case.  This bug is also directly affecting the development of opensolaris.org.
                                     

PUBLIC COMMENTS

From email conversation with Xuelei Fan:

Xuelei Fan wrote:

> As the bug description, the SSL server side want to check the client 
> hostname against the client certificate(I call this as client identity 
> checking), and broken at 6u10. However, by default, the SSL server side 
> does not enable the client identity checking, so I think we need more 
> information about the user's application code, I suppose the user call 
> sun internal API, and enabled the client identity checking.

No, it calls setNeedClientAuth(true) on the SSLServerSockets it creates by subclassing X509ExtendedTrustManager.  The code in question can be seen at http://auth.opensolaris.org/svn/showfile.svn?path=/trunk/AuthSSL/src/org/opensolaris/auth/ssl/SSLSocketManager.java&revision=HEAD&name=auth

I raised a number of bugs against this area last year, for reference they are:

6586258 X509ExtendedTrustManager is in the wrong namespace       
http://monaco.sfbay/detail.jsf?cr=6586258
                                           
6586274 SSLSocketFactory and SSLServerSocketFactory can't be configured                                     
http://monaco.sfbay/detail.jsf?cr=6586274

6586276 SSLSockets and SSLEngines need a switch to enable hostname validation
http://monaco.sfbay/detail.jsf?cr=6586276

> I don't know whether or not you have approach to the application 
> programmer, I want to know whether or not their codes call the sun 
> internal API, SSLEngineImpl.trySetHostnameVerification() or 
> SSLSocketImpl.trySetHostnameVerification(), at the server accepted 
> sockets. If it is an essential call, I would like to know why, and would 
> try to make improvement.

No, neither of those APIs are called.

> At 6u10, we made a few changes on the way to get a ssl socket hostname 
> (fixed for bug 6614957). Before 6u10, the socket name will be resolved 
> from domain name services, from 6u10, the socket name will not be 
> resolved any more. For this case, the SSL server accepted socket's 
> hostname will be resolved from 127.0.0.1 to "localhost" before 6u10, but 
> it will not changed at 6u10. So, even the application enabled the client 
> identity checking, the checking will pass at 6u7, for "localhost" is an 
> subject alternative name of the client certificate.

The problem is that whatever you did for that fix has broken another part of the JDK, this from the stack trace I included in the original bug report:

/sun/security/util/HostnameChecker.java

    /**
     * Perform the check.
     *
     * @exception CertificateException if the name does not match any of
     * the names specified in the certificate
     */
    public void match(String expectedName, X509Certificate cert)
            throws CertificateException {
        if (isIpAddress(expectedName)) {
           matchIP(expectedName, cert);
        } else {
           matchDNS(expectedName, cert);
        }
    }

Now, because expectedName is *always* an IP address in 1.6.1_10, the first branch of the check will *always* be taken, and unless there is a Subject Alternative Name IP entry in the cert, the the check will *always* fail.  If you've removed the reverse DNS lookup elsewhere then you've broken this method as a result.  I can't drill down any further into the stack trace because I don't have access to the source of the com.sun.net.ssl.internal.ssl.* classes that call HostnameChecker.match.

Note that in general Subject Alternative Names use the DNS name rather than IP addresses, because it means the certificate is valid even if the service is moved to another IP or DNS round-robin'ed across several IPs.

> Typically, the server has no external knowledge of what the client's 
> identity ought to be and so checks are not possible. However, if the 
> server understand  how to check the client's identity, a proper 
> workaround is to customize the X509ExtendedTrustManager.checkClientTrust().

I disagree, client authentication is a defined part of the specification, and is often used where the communicating parties are both applications.

Also, I assume you mean 'X509ExtendedTrustManager.checkClientTrusted()'.  Overriding checkClientTrusted is exactly what the code in question does, and that's exactly what is now broken in 1.6.0_10.