EVALUATION

Even though there is no way to request it through the shm* apis, solaris with ISM enables execute permissions on large page memory, at least on solaris 10.  Using a JVM where can_execute_large_page_memory() returns true on solaris with ISM and the code cache is mapped at 0xfb800000:

   $ pmap -s 10827 | egrep FB800000
   FB800000   32768K   4M rwxsR    [ ism shmid=null ]

However, I was not able to test on solaris 8, the only release where ISM should be used, and the execute permissions are not documented.  So can_execute_large_page_memory() returns false on solaris with ISM.
                                     

EVALUATION

Other structures that may use large pages (the marking bitmap and region table used by the parallel compacting collector) must also set parameter large == true when constructing the ReservedSpace.
                                     

SUGGESTED FIX

Add a new method os::can_execute_large_page_memory() that returns true if large page memory can have execute permissions.  Returns false on linux and solaris with ISM.

Change the code cache creation code to use large pages only if can_execute_large_page_memory() returns true.

Initialize the os::_page_sizes array on solaris when ISM is used.

Change windows os::reserve_memory_special() to include execute permission.

Call the ReservedSpace ctor with parameter large == true when requesting large pages
for the code cache and other non-java-heap data structures.

--- old/src/share/vm/runtime/os.hpp	Mon Mar 24 18:35:15 2008
+++ new/src/share/vm/runtime/os.hpp	Mon Mar 24 18:35:14 2008
@@ -236,6 +236,7 @@
   static bool   large_page_init();
   static size_t large_page_size();
   static bool   can_commit_large_page_memory();
+  static bool   can_execute_large_page_memory();
 
   // OS interface to polling page
   static address get_polling_page()             { return _polling_page; }
--- old/src/os/linux/vm/os_linux.cpp	Mon Mar 24 18:35:16 2008
+++ new/src/os/linux/vm/os_linux.cpp	Mon Mar 24 18:35:16 2008
@@ -2486,6 +2486,10 @@
   return false;
 }
 
+bool os::can_execute_large_page_memory() {
+  return false;
+}
+
 // Reserve memory at an arbitrary address, only if that area is
 // available (and not reserved for something else).
 
--- old/src/os/solaris/vm/os_solaris.cpp	Mon Mar 24 18:35:19 2008
+++ new/src/os/solaris/vm/os_solaris.cpp	Mon Mar 24 18:35:19 2008
@@ -3067,6 +3067,8 @@
   if (UseISM) {
     // ISM disables MPSS to be compatible with old JDK behavior
     UseMPSS = false;
+    _page_sizes[0] = _large_page_size;
+    _page_sizes[1] = vm_page_size();
   }
 
   UseMPSS = UseMPSS &&
@@ -3155,6 +3157,10 @@
 bool os::can_commit_large_page_memory() {
   return UseISM ? false : true;
 }
+
+bool os::can_execute_large_page_memory() {
+  return UseISM ? false : true;
+}
 
 static int os_sleep(jlong millis, bool interruptible) {
   const jlong limit = INT_MAX;
--- old/src/os/windows/vm/os_windows.cpp	Mon Mar 24 18:35:21 2008
+++ new/src/os/windows/vm/os_windows.cpp	Mon Mar 24 18:35:21 2008
@@ -2523,9 +2523,13 @@
   return false;
 }
 
+bool os::can_execute_large_page_memory() {
+  return true;
+}
+
 char* os::reserve_memory_special(size_t bytes) {
   DWORD flag = MEM_RESERVE | MEM_COMMIT | MEM_LARGE_PAGES;
-  char * res = (char *)VirtualAlloc(NULL, bytes, flag, PAGE_READWRITE);
+  char * res = (char *)VirtualAlloc(NULL, bytes, flag, PAGE_EXECUTE_READWRITE);
   return res;
 }
 
--- old/src/share/vm/memory/heap.cpp	Mon Mar 24 18:35:24 2008
+++ new/src/share/vm/memory/heap.cpp	Mon Mar 24 18:35:23 2008
@@ -102,8 +102,9 @@
   _log2_segment_size = exact_log2(segment_size);
 
   // Reserve and initialize space for _memory.
-  const size_t page_size = os::page_size_for_region(committed_size,
-                                                    reserved_size, 8);
+  const size_t page_size = os::can_execute_large_page_memory() ?
+          os::page_size_for_region(committed_size, reserved_size, 8) :
+          os::vm_page_size();
   const size_t granularity = os::vm_allocation_granularity();
   const size_t r_align = MAX2(page_size, granularity);
   const size_t r_size = align_size_up(reserved_size, r_align);
@@ -111,7 +112,7 @@
 
   const size_t rs_align = page_size == (size_t) os::vm_page_size() ? 0 :
     MAX2(page_size, granularity);
-  ReservedSpace rs(r_size, rs_align, false);
+  ReservedSpace rs(r_size, rs_align, rs_align > 0);
   os::trace_page_sizes("code heap", committed_size, reserved_size, page_size,
                        rs.base(), rs.size());
   if (!_memory.initialize(rs, c_size)) {
--- old/src/share/vm/gc_implementation/parallelScavenge/parMarkBitMap.cpp	Mon Mar 24 18:35:25 2008
+++ new/src/share/vm/gc_implementation/parallelScavenge/parMarkBitMap.cpp	Mon Mar 24 18:35:25 2008
@@ -41,7 +41,7 @@
 
   const size_t rs_align = page_sz == (size_t) os::vm_page_size() ? 0 :
     MAX2(page_sz, granularity);
-  ReservedSpace rs(bytes, rs_align, false);
+  ReservedSpace rs(bytes, rs_align, rs_align > 0);
   os::trace_page_sizes("par bitmap", raw_bytes, raw_bytes, page_sz,
                        rs.base(), rs.size());
   _virtual_space = new PSVirtualSpace(rs, page_sz);
--- old/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.cpp	Mon Mar 24 18:35:27 2008
+++ new/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.cpp	Mon Mar 24 18:35:27 2008
@@ -413,7 +413,7 @@
 
   const size_t rs_align = page_sz == (size_t) os::vm_page_size() ? 0 :
     MAX2(page_sz, granularity);
-  ReservedSpace rs(bytes, rs_align, false);
+  ReservedSpace rs(bytes, rs_align, rs_align > 0);
   os::trace_page_sizes("par compact", raw_bytes, raw_bytes, page_sz, rs.base(),
                        rs.size());
   PSVirtualSpace* vspace = new PSVirtualSpace(rs, page_sz);
                                     

WORK AROUND

Do not use large pages on windows (omit the option -XX:+UseLargePages).  This reduces performance on large heaps, so is not acceptable for many users.
                                     

EVALUATION

The fix for 6588638 (improve support for large pages) generalized large page support and started mapping the code cache with large pages in addition to the java heap.  This did not work on systems where large page memory cannot be committed on demand.  Solaris MPSS (the default and preferred method on solaris 9 and later) allows large page memory to be committed on demand and works fine.

Linux and solaris with ISM use shared memory segments to get large pages; the memory cannot be committed on demand.  Windows uses VirtualAlloc() to get large pages; the memory cannot be committed on demand. 

There are multiple problems.  The first is that the hotspot ReservedSpace ctor takes a boolean parameter, large, which must be set to true to get large page memory on systems that cannot commit large pages on demand.  The code cache creation code was not passing large == true to the ReservedSpace ctor when attempting to get large pages.

The second is that the code cache memory must have execute permissions.  On windows, the code to reserve large page memory was not requesting execute permissions.  On linux and solaris + ISM, the shm* apis used to request large page memory do not allow the execute bit to be specified, so the code cache cannot be mapped using large pages on these systems.

The final problem, specific to solaris + ISM, is that the JVM's _page_size_table was not being initialized properly.

Created RFE 6677036 to track enabling large pages for the code cache on linux, when/if the kernel allows large page memory to have execute permission.