EVALUATION
I created a script that invokes the CreateSymbols annotation processor 100 times and report any failures during execution. I found out that, out of 100 runs, there were around 10 failures (the exact number of failures varies with time).
If the script was modified so that the new hash changes were disabled (by setting the system property Djdk.map.althashing.threshold=-1) the failures disappeared. This means the failure must be related to the hash changes. I then looked closer at the code in CreateSymbols.java, and found that the call to JavaFileManager.list() was not indexing all files in the input archives in a consistent order - more specifically, the order was changing each time CreateSymbols was executed (which is a very typical side-effects of the new hash changes).
If JavaFileManager.list() is modified so that it returns files listed in a consistent, non-random order - the issue goes away (i.e. no failures out of 100 runs). Of course this is only a temporary workaround that will buy us some time in order to figure out what the underlying issue exactly is - but at least it should be enough to get rid of the random build failures that have been seen recently.
|