SUGGESTED FIX

------- FileChannelImpl.c -------
235a236,241
> #ifdef __solaris__
> typedef struct flock64 FLOCK;
> #else
> typedef struct flock FLOCK;
> #endif
> 
244,246c250,251
<     struct flock64 fl;
< 
<     fl.l_whence = SEEK_SET;
---
>     FLOCK fl;
>     #ifdef __solaris__
248a254,262
>     #else
>     if (size > 0x7fffffff) { 
>         size = 0x7fffffff;
>     } 
>     fl.l_len = (off_t)size;
>     fl.l_start = (off_t)pos;
>     #endif
> 
>     fl.l_whence = SEEK_SET;
259a274,275
>       fprintf(stderr, "block=%d, pos=%d, size=%d, shared=%d, ret=%d, err=%d, f
l=%d\n",
>               block, (int)pos, (int)size, shared, lockResult, errno, sizeof(fl
.l_len));
263c279
<       if (errno == EINTR)
---
>         if (errno == EINTR)
277c293
<     struct flock64 fl;
---
>     FLOCK fl;
280c296
<     fl.l_whence = SEEK_SET;
---
>     #ifdef __solaris__
282a299,307
>     #else
>     if (size > 0x7fffffff) {
>         size = 0x7fffffff;
>     }
>     fl.l_len = (off_t)size;
>     fl.l_start = (off_t)pos;
>     #endif
> 
>     fl.l_whence = SEEK_SET;
288a314
>
                                     

EVALUATION

I was able to reproduce the problem on a JDS box. 
When the command "strace java Lock" was executed, the output showed that only F_SETLKW was passed instead of F_SETLKW64. 
fcntl64(3, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
Where as with JDK 1.5 build, when "strace java Lock" was executed, F_SETLKW64 was called. And hence the testcase was not failing. 
fcntl64(3, F_SETLKW64, {type=F_WRLCK, whence=SEEK_SET, start=0, len=10}, 0xbfffd2dc) = 0
The difference is in the definition of fcntl.h file in both the cases. 
The differences in fcntl.h file are 
#define F_SETLKW64 7 
and #define F_SETLKW64 35
The problem can be due to the fact that the build machines differ from 1.4 to 1.5.
We need to use a proper 64-bit machine to build this, since the changes from 1.4.2_09 to 1.4.2_10 included a structure change from 32-bit to 64-bit. When these changes are buit on a 64-bit machine, the problem will not occur. 

Thanks,
Vidya.

The suggested fix for the bug is causing a failure of the customer's fix. 
The evaluation done so far is, the tryLock() function should return -1 or E_AGAIN
The execption is thrown only in this case. But in the regression test case, exception is not thrown. E_AGAIN error is not thrown. The only case when we return null, is the lock is spread across two processes. 

The fcntl64 function can be used when the macro __USE_LARGEFILE64 is defined.
But the testcase is failing even then on the SLES machine. The use of !defined(_LP64) also was breaking the customer fix.. 

We should evaluate if it is the problem with the testcase.