This appears to be a longstanding problem with x64 port. The cmpfp_fixup code is trying to fixup the condition codes when NaN is encountered but instead of emitting addq (%rsp),0xffffff2b it's emitting addq %rsp, 0xffffff2b which changes the stack and then pops the garbage of some kind into the flags register.
0xfffffd7ff9511c61: ucomisd 0xfffffffffffffa1f [ 0xfffffffffffffa1f ](%rip),%xmm0
0xfffffd7ff9511c69: jnp 0xfffffd7ff9511c78 [ 0xfffffd7ff9511c78, .+0xf ]
0xfffffd7ff9511c70: andq $0xffffffffffffff2b,%rsp
0xfffffd7ff9511c79: cmovl.ne 0x0000000000000044(%rsp),%ecx
7056380: VM crashes with SIGSEGV in compiled code
Summary: code was using andq reg, imm instead of addq addr, imm
Reviewed-by: kvn, jrose, twisti
In the changes for 6961690 a copy of cmpfp_fixup was moved inline but
was translated incorrectly so that it was and'ing rsp instead of
(rsp). This would cause garbage to be popped into the flags and
corrupt rsp. Depending on the OS and values involved you would die at
the next of use the flags or later after a return. The fix is to use
the right andq form. I also converted the cmpfp_fixup code into
MacroAssembler so that it was obviously equivalent. Tested with
failing test case and by inspection of the resulting assembly.