When inflating a file whose inflated size is 2GB or more, GZIPInputStream
incorrectly throws IOException.

The test program below compresses the specified number of double-words, then
decompresses and verifies.   Examples:

$ java -server GzipTest file-64KB.gz 8192
File file-64KB.gz written.

$ java -server GzipTest file-2GB.gz 268435456
File file-2GB.gz written.
Exception in thread "main" java.lang.Error: Error reading
	at GzipTest.verify(GzipTest.java:82)
	at GzipTest.main(GzipTest.java:26)
Caused by: java.io.IOException: Corrupt GZIP trailer
	at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:174)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:89)
	at GzipTest.verify(GzipTest.java:67)
	... 1 more


$ cat GzipTest.java
import java.io.*;
import java.nio.ByteBuffer;
import java.util.zip.*;

public class GzipTest {

    private static final byte[] PI;
    static {
        ByteBuffer bb = ByteBuffer.allocate(8);
        bb.putDouble(Math.PI);
        PI = bb.array();
    }

    private final File file;
    private final int count;

    public static void main(String[] args) {
        if (args.length != 2) {
            System.err.println("Usage: GzipTest <filename.gz> <count>");
            System.exit(1);
        } 
        File outputFile = new File(args[0]);
        GzipTest test = new GzipTest(outputFile, Integer.parseInt(args[1]));
        test.write();
        System.out.println("File " + args[0] + " written.");
        if (test.verify()) {
            System.err.println("Verification FAILED!");
        }
    }

    public GzipTest(File file, int count) {
        this.file = file;
        this.count = count;
    }

    /** Writes count copies of PI to a compressed file. */
    public void write() {
        GZIPOutputStream gzipper;
        try {
            gzipper = new GZIPOutputStream(new FileOutputStream(file));
        } catch (IOException error) {
            throw new IllegalArgumentException("Error opening " + file);
        }
        try {
            for (int i = 0; i < count; i++) {
                gzipper.write(PI, 0, 8);
            }
            gzipper.close();
        } catch (IOException e) {
            throw new Error("Error writing", e);
        }
    }

    /** Returns true on error. */
    public boolean verify() {
        GZIPInputStream gunzipper;
        try {
            gunzipper = new GZIPInputStream(new FileInputStream(file));
        } catch (IOException error) {
            throw new IllegalArgumentException("Error opening " + file);
        }
        byte[] input = new byte[8];
        try {
            for (int i = 0; true; i++) {
                int pos = 0;
                do {
                    int read = gunzipper.read(input, pos, input.length - pos);
                    if (read < 0) {
                        gunzipper.close();
                        return i != count;
                    }
                    pos += read;
                } while (input.length - pos > 0);
                for (int j = 0; j < input.length; j++) {
                    if (input[j] != PI[j]) {
                        gunzipper.close();
                        return true;
                    }
                } 
            }     
        } catch (IOException e) {
            throw new Error("Error reading", e);
        }
    }
}
Posted Date : 2006-01-23 11:04:18.0

N/A

Here is a somewhat more minimal test case:

----------------------------------------------------------------------------
import java.io.*;
import java.nio.ByteBuffer;
import java.util.zip.*;

public class GzipTest {
    static void readFully(InputStream s, byte[] input) throws Exception {
	int pos = 0;
	int n;
	while ((n = s.read(input, pos, input.length-pos)) > 0)
	    pos += n;
	if (pos != input.length)
	    throw new Exception("Unexpected EOF");
    }

    public static void main(String[] args) throws Exception {
        if (args.length != 2)
	    throw new Exception("Usage: GzipTest FILENAME COUNT");

	File file = new File(args[0]);
	int count = Integer.parseInt(args[1]);

	ByteBuffer bb = ByteBuffer.allocate(8);
        bb.putDouble(Math.PI);
        byte[] PI = bb.array();

	// Write count copies of PI to a compressed file
	GZIPOutputStream out = new GZIPOutputStream(new FileOutputStream(file));
	for (int i = 0; i < count; i++)
	    out.write(PI, 0, 8);
	out.close();

	// Verify file
	GZIPInputStream in = new GZIPInputStream(new FileInputStream(file));
	byte[] maybePI = new byte[8];
	for (int i = 0; i < count; i++) {
	    readFully(in, maybePI);
	    for (int j = 0; j < PI.length; j++)
		if (maybePI[j] != PI[j])
		    throw new Exception("data corruption");
	}
	if (in.read(maybePI, 0, 1) > 0)
	    throw new Exception("Unexpected NON-End of File");
    }
}
----------------------------------------------------------------------------
(  xxxxx@xxxxx  ) ~/src/toy/5092263 $ jver 1.5 java -server GzipTest foo 268435456
Exception in thread "main" java.io.IOException: Corrupt GZIP trailer
	at java.util.zip.GZIPInputStream.readTrailer(GZIPInputStream.java:175)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:89)
	at GzipTest.main(GzipTest.java:41)

The Gzip file format uses 4 byte fields for byte counts,
so those fields are only accurate modulo 2**32.

gzip 1.2.4 does not support files larger than 4GB.  You need
the (currently beta) gzip 1.3 for that.

  xxxxx@xxxxx   2004-08-29

It's interesting to note that gzip had essentially the same bug
as the jdk, both for many years.
  xxxxx@xxxxx   2005-2-15 19:21:30 GMT

This also exists in Java 1.3.1, Java 1.4.X

Command-line gzip handles the overflow in a way that doesn't corrupt/end the decompression -- java should be able to do the same.

See also bug 4262583  -- which claims this is fixed in 1.5. However, we've seen it in 1.5 so whoever closed 4262583  did so in error.