Customers are experiencing diminished throughput in Access Manager / OpenSSO products on T51xx / T52xx machines (high concurrency). CPU util on those machines are capped at 10~20%.

All thread dumps, generated with kill -3, show on average 80 threads waiting on a lock to be released in :

javax.security.auth.login.Configuration.getConfiguration

The stack trace below shows the thread that currently has the lock. Most of the other worker threads are waiting for a monitor entry, as shown below :

"service-j2ee" daemon prio=3 tid=0x01e7b060 nid=0x137 waiting for monitor entry [0x32d0d000..0x32d0f7d8]
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:190)
        - waiting to lock <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AMLoginContext.getModuleFromAuthConfiguration(AMLoginContext.java:1804)

I saw a bug / fix regarding the PolicyFile.implies call, but moving to update 19 doesn't seem to change anything.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6552236

It looks to me that the synchronized getConfiguration() is a major bottleneck in highly multi-threaded environment.


"service-j2ee" daemon prio=3 tid=0x0294ac88 nid=0x12f runnable [0x32f8d000..0x32f8f7d8]
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.provider.PolicyFile.addPermissions(PolicyFile.java:1370)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1325)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1291)
        at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1227)
        at sun.security.provider.PolicyFile.implies(PolicyFile.java:1182)
        at com.sun.enterprise.security.provider.PolicyWrapper.implies(PolicyWrapper.java:135)
        at java.security.ProtectionDomain.implies(ProtectionDomain.java:195)
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:249)
        at java.security.AccessController.checkPermission(AccessController.java:427)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
        at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:192)
        - locked <0xec228f50> (a java.lang.Class)
        at com.sun.identity.authentication.service.AuthUtils.isPureJAASModulePresent(AuthUtils.java:1399)
        at com.sun.identity.authentication.service.AMLoginContext.executeLogin(AMLoginContext.java:299)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:327)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:263)
        at com.sun.identity.authentication.server.AuthContextLocal.login(AuthContextLocal.java:183)
        at com.sun.identity.authentication.UI.LoginViewBean.getLoginDisplay(LoginViewBean.java:792)
        at com.sun.identity.authentication.UI.LoginViewBean.processLoginDisplay(LoginViewBean.java:1034)
        at com.sun.identity.authentication.UI.LoginViewBean.processLogin(LoginViewBean.java:734)
        at com.sun.identity.authentication.UI.LoginViewBean.forwardTo(LoginViewBean.java:444)
        at com.iplanet.jato.ApplicationServletBase.dispatchRequest(ApplicationServletBase.java:981)
Posted Date : 2009-05-26 17:24:21.0

See related CR filed for Access Manager : CR 6844567.
Posted Date : 2009-05-26 17:24:21.0

N/A

Sean Mullan started looking at this issue. He suggested the following fix :

One simple fix is to change Configuration.getConfiguration so that the SecurityManager check is not done inside the lock. This is not necessary and could alleviate the problem by not locking the method while all of the policy processing is done.

1) remove synchronized keyword from getConfiguration

2) add synchronized block after the SecurityManager.checkPermission call.
Posted Date : 2009-05-27 14:30:24.0

AFter using Sean's modified Configuration class for JDK 1.5.0_19, throughput was much higher (10 to 20% gain) and CPU util is now around 29% (compared to 20%).

There are no more threads waiting to on a monitor entry (waiting to lock) for Configuration.getConfiguration.
Posted Date : 2009-05-28 01:08:00.0