EVALUATION

I offered to help Vinnie with this, but haven't been able to figure this out.

My original thought was that Windows 7 might have been enhanced so that Ciphers can take either public or private keys for encryption, but that is not the case. I was able to get certain PrivateKeys to initialize a Cipher object on XP and Windows 7.

Here is a small code snippet that shows a simplified test case.

import java.security.*;
import java.util.Enumeration;
import javax.crypto.Cipher;

public class Main {

    public static void main(String[] args) throws Exception {

        Provider[] p = Security.getProviders();

        KeyStore ks = KeyStore.getInstance("Windows-MY", "SunMSCAPI");
        ks.load(null, null);

        Enumeration e = ks.aliases();

        PrivateKey privateKey = null;

        PublicKey publicKey = null;

        while (e.hasMoreElements()) {
            String alias = (String) e.nextElement();
            if (ks.isKeyEntry(alias)) {
                System.out.println("got it: " + alias);
                privateKey = (PrivateKey) ks.getKey(alias, null);
                publicKey = (PublicKey) ks.getCertificate(alias).getPublicKey();
            }
        }

        System.out.println(privateKey);

        Signature sig = Signature.getInstance("NONEwithRSA", "SunMSCAPI");
        sig.initSign(privateKey);
        sig.update((byte)0x15);
        byte [] ba = sig.sign();
    }
}

For creating keypairs/certs, I used one of three methods:

1)  The MS SDK makecert utility.  (FAIL)

% makecert c:/tmp/cert.cer -a sha1 \
    -n "CN=Sun JSSE TestService, O=Java, C=US" -ss My

2)  keytool  (PASS)

[wetmore@dualcongas] 338 >keytool -genkey -keyalg rsa -keystore foo.p12 -storety
pe pkcs12
Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:  Sun JSSE TestService
What is the name of your organizational unit?
  [Unknown]:  Java
What is the name of your organization?
  [Unknown]:  Sun
What is the name of your City or Locality?
  [Unknown]:  SCA
What is the name of your State or Province?
  [Unknown]:  CA
What is the two-letter country code for this unit?
  [Unknown]:  US
Is CN=Brad, OU=Java, O=Sun, L=SCA, ST=CA, C=US correct?
  [no]:  yes

3) The interoperability test suite test certs that were created for TLS 1.2 during JDK 7 development.  (PASS)

% cd <JDK7>/jdk/test/closed/sun/security/ssl/sanity/compatibility/helpers/certs
% sh ./generate.sh

I thought maybe it maybe had something to do with  keyusages, but 2) doesn't produce keyusages, so I'm not seeing something.

Only 1) failed to run, it failed with:

[wetmore@dualcongas] 317 >java Main
got it: Sun JSSE Test Service
RSAPrivateKey [size=1024 bits, type=Signature, container=JoeSoft]
Exception in thread "main" java.security.ProviderException: java.security.KeyExc
eption: Bad Key.

        at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:242)
        at sun.security.mscapi.RSACipher.engineDoFinal(RSACipher.java:266)
        at javax.crypto.Cipher.doFinal(DashoA13*..)
        at java.security.Signature$CipherAdapter.engineSign(Signature.java:1225)

        at java.security.Signature$Delegate.engineSign(Signature.java:1128)
        at java.security.Signature.sign(Signature.java:522)
        at Main.main(Main.java:37)
Caused by: java.security.KeyException: Bad Key.

        at sun.security.mscapi.RSACipher.encryptDecrypt(Native Method)
        at sun.security.mscapi.RSACipher.doFinal(RSACipher.java:222)
        ... 6 more

To do cert management while testing, it is best to use certmgr rather than using the cert manager in IE.  If you use IE, go into the Internet Options, then content, certificates, personal certificates.

Originally, I was thinking that Ciphers wouldn't allow initialization with PrivateKeys, so we might want to consider rewriting this using CryptSignMessage, if there is an option for NoPadding.  That is, we simply have a regular Cipher *AND* Signature.  But at this point, I had no problems using either 2)/3), so I'm not sure where the problem really lies.

Also, when looking at this, please note the second webbug, I think the problems described are different, and should be investigated further.  This seems to be true, I was getting different values on each run.

    public static void main(String[] args) throws Exception {

        Provider[] p = Security.getProviders();

        KeyStore ks = KeyStore.getInstance("Windows-MY", "SunMSCAPI");
        ks.load(null, null);

        Enumeration e = ks.aliases();

        PrivateKey privateKey = null;

        PublicKey publicKey = null;

        while (e.hasMoreElements()) {
            String alias = (String) e.nextElement();
            if (ks.isKeyEntry(alias)) {
                System.out.println("got it: " + alias);
                privateKey = (PrivateKey) ks.getKey(alias, null);
                publicKey = (PublicKey) ks.getCertificate(alias).getPublicKey();
            }
        }

        System.out.println(privateKey);

        Signature sig1 = Signature.getInstance("NONEwithRSA", "SunMSCAPI");
        sig1.initSign(privateKey);
        sig1.update((byte)0x15);
        byte [] ba = sig1.sign();
        System.out.println("Output Length: " + ba.length*8);
        for (byte b : ba) {
            System.out.print(b + " ");
        }
        System.out.println();

        Signature sig2 = Signature.getInstance("NONEwithRSA", "SunMSCAPI");
        sig2.initSign(privateKey);
        sig2.update((byte)0x15);
        byte [] ba2 = sig2.sign();
        System.out.println("Output Length: " + ba2.length*8);
        for (byte b : ba2) {
            System.out.print(b + " ");
        }
        System.out.println();

        // using the first available
        sig1 = Signature.getInstance("NONEwithRSA");
        sig1.initVerify(publicKey);
        sig1.update((byte)0x15);
        if (sig1.verify(ba)) {
            System.out.println("Verify PASSED");
        } else {
            System.out.println("Verify FAILED");
        }

        sig1 = Signature.getInstance("NONEwithRSA");
        sig1.initVerify(publicKey);
        sig1.update((byte)0x15);
        if (sig1.verify(ba2)) {
            System.out.println("Verify PASSED");
        } else {
            System.out.println("Verify FAILED");
        }
    }
}

which outputs:

[wetmore@dualcongas] 330 >java Main
got it: CN=Brad,OU=Java,O=Sun,L=SCA,ST=CA,C=US
RSAPrivateKey [size=1024 bits, type=Exchange, container={A18137D3-228C-42C9-A81C
-AC29F1AC4CA7}]
Output Length: 1024
59 33 -5 -115 103 -97 -123 -41 38 -93 53 -52 20 39 -51 -114 97 39 53 -1 32 48 32
 -20 28 2 89 113 123 -84 -31 -68 -24 -107 -93 66 -76 -80 -47 -81 -91 79 115 -35
110 -51 -44 94 -112 114 -126 -49 -127 38 -61 -48 -89 -28 57 -74 -117 95 49 44 -4
2 -84 27 94 -90 -33 23 -7 -35 -63 -35 -15 -59 66 16 -15 12 105 -106 -86 17 122 -
4 121 -87 40 -63 -121 119 4 -3 17 112 50 -120 72 -35 -12 10 61 -34 84 111 3 -59
50 103 88 118 -102 -37 -24 65 3 -6 -3 -31 -80 -74 119 -59 -35 -119 -97
Output Length: 1024
102 62 11 -53 -78 -31 -41 -114 -127 70 -78 17 84 71 16 -32 125 27 -75 -94 -50 -2
9 -51 -98 106 39 -92 113 -78 -99 -107 -113 12 -39 98 -19 -68 -91 -6 84 99 104 -1
23 -77 -123 -52 -45 83 -6 118 21 -56 58 92 -80 27 -62 12 -12 34 -28 -24 -96 -38
73 -118 -54 35 -20 -64 -76 -26 -10 -96 77 -72 44 -74 -89 69 -91 13 -122 -89 -22
-64 72 -23 28 -11 84 -62 -90 -85 -91 -80 60 77 -55 114 -34 -10 105 -53 -66 -122
-82 -66 12 -96 63 -65 -1 -49 89 -45 -48 -93 -108 94 -120 103 22 53 -25 69 102 -2
3
Verify FAILED
Verify FAILED

Note the signatures are indeed different.

At this point, I need to move on.