=20
To establish an encrypted network connection for _interactive_
use you need to compress the stream _before_ encryption because
an already encrypted stream is not compressible.

Today it is not possible to reuse java.util.zip for compression
because the Java VM calls deflate() in the libz with
Z_NO_FLUSH (and Z_FINISH for EOF) only. This means that in most
cases the deflate() does not emit enough data for the
decompressor to reassemble the complete data packet because
it waits for more input.

This is what the deflate() parameter Z_SYNC_FLUSH is for.
It enables the inflate() of the receiver to create the full
data packet. This will decrease the compression ration slightly=20
but will allow to reuse java.util.zip for interactive use.

Please note that this is _not_ Z_FULL_FLUSH, which would tell
deflate() to do a full reset of the compressor.
(Review ID: 52795)
======================================================================




N/A API deficiency
The java.util.zip routines do not support the zlib functionality of
Z_SYNC_FLUSH. This makes it impossible to write a client-server application
where the client and server pass each other messages along a compressed stream,
as you need to be able to compress a message (given the current dictionary, not
just starting with a new one), flush and byte-align the output, and then send
the message across the socket. This is possibile with the c implemention of
zlib, as the Z_SYNC_FLUSH will flush and byte-align the output of the current
compression operation.
(Review ID: 97225)
======================================================================




C:\>java -version
java version "1.2"
Classic VM (build JDK-1.2-V, native threads)


I am trying to develop a client/server application using sockets. To improve
performance I am trying to zip de data sent to the socket, using
java.util.zip.DeflaterOutputStream and java.util.zip.inflaterInputStream, but
the application stops when the client or the server write something in the
output stream.

Without zip stream it works fine.

Here is the code of client:

import java.net.*;
import java.io.*;
import java.util.zip.*;

public class SocketClientApp
{
  public static void main( String [] args ) throws IOException
  {
    Socket clientSocket = null;
    PrintWriter out     = null;
    BufferedReader in   = null;

    try
    {
      clientSocket = new Socket( "localhost", 4444 );
      out = new PrintWriter( new DeflaterOutputStream(
                                 clientSocket.getOutputStream() ), true );
      in  = new BufferedReader( new InputStreamReader( new InflaterInputStream(
                                 clientSocket.getInputStream() ) ) );

      BufferedReader stdIn = new BufferedReader( new InputStreamReader(
                                                               System.in ) );

      String fromServer, fromUser;
      while( ( fromServer = in.readLine() ) != null )
      {
        System.out.println( "Server said: " + fromServer );
        fromUser = stdIn.readLine();
        if( fromUser != null )
          out.println( fromUser );

        if( fromUser.equals( "TChau!" ) )
          break;
      }

      out.close();
      in.close();
      stdIn.close();
      clientSocket.close();
    }
    catch( UnknownHostException e)
    {
      e.printStackTrace();
    }
    catch( IOException e )
    {
      e.printStackTrace();
    }
  }
}

Here is the code of the server Thread:

import java.net.*;
import java.io.*;
import java.util.zip.*;

public class SocketServerThread extends Thread
{
   private Socket clientSocket;

   public SocketServerThread( Socket clientSocket )
   {
      super( "MultiServerThread" );

      this.clientSocket = clientSocket;
   }

   public void run()
   {
      try
      {
         PrintWriter out   = new PrintWriter( new DeflaterOutputStream(
                                      clientSocket.getOutputStream() ), true );
         BufferedReader in = new BufferedReader( new InputStreamReader( new
InflaterInputStream( clientSocket.getInputStream() ) ) );

         String inputLine, outputLine;

         outputLine = "Server: Hello!";
         out.println( outputLine );

         while( ( inputLine = in.readLine() ) != null )
         {
            System.out.println( "Received: " + inputLine );
            if( inputLine.equals( "TChau!" ) )
              break;
            outputLine = "You typed: " + inputLine;
            out.println( outputLine );
         }

         in.close();
         out.close();
         clientSocket.close();
      }
      catch( IOException e )
      {
         e.printStackTrace();
      }
   }
}

Here is the server application:

import java.net.*;
import java.io.*;

public class MultiSocketServerApp
{
   static public void main( String [] args ) throws IOException
   {
      ServerSocket serverSocket = null;
      try
      {
        serverSocket = new ServerSocket( 4444 );
      }
      catch( IOException e )
      {
        e.printStackTrace();
      }

      while( true )
      {
        Socket clientSocket = null;
        try
        {
           clientSocket = serverSocket.accept();
           SocketServerThread newThread = new SocketServerThread(
clientSocket );
           newThread.start();
        }
        catch( IOException e )
        {
           e.printStackTrace();
           System.exit(-1);
        }
      }
   }
}
(Review ID: 98981)
======================================================================




java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C)
Java HotSpot(TM) Client VM (build 1.3.0-C, mixed mode)

There is no way to do a partial or sync flush of java.util.zip.Deflater. A
partial flush is required when you want to send multiple compressed segments on
a single stream such that they use the same compression parameters and
dictionary. In particular, it is needed to properly implement the compression
mode of Secure Shell.

Add a deflate() method with the following signature:
  public synchronized int deflate( byte[] b, int off, int len, int flush );

where flush is one of
  public static final int DEFLATER_NO_FLUSH      = 0;
  public static final int DEFLATER_PARTIAL_FLUSH = 1;
  public static final int DEFLATER_SYNC_FLUSH    = 2;
  public static final int DEFLATER_FULL_FLUSH    = 3;
  public static final int DEFLATER_FINISH        = 4;

You can make this change without breaking any existing code: when Deflater's
private field 'finish' is true, the existing deflate() methods should call the
new method with a flush argument of DEFLATER_FINISH. Otherwise they would
specify DEFLATER_NO_FLUSH.

It's actually a bit more complicated than that. If the caller supplies
DEFLATER_FINISH, you would have to call end() to update the 'finish' field.
(Review ID: 115912)
======================================================================

=20
Port a compressor/decompressor to Java and use that instead
of java.util.zip. To avoid porting to several platforms this
must be done in 100% Java which will decrease the compressor
speed.
======================================================================

A flush with Z_SYNC_FLUSH is all we need.
Posted Date : 2009-02-20 07:53:03.0

Someone wrote zlib in 100% Java ? I only found versions in other languages, but
not in 100% Java

Is this bug related to 4077821?

I encountered this problem recently and scratched my head for a long time
before finding  this bug report. We have developed a workaround in the form of
a serialized class that has a byte[] member variable and methods to serialize,
compress, decompress, and deserialize into the array.

So, for example, the server can take an object, serialize it into the array,
compress the array, and send the whole thing down the socket.  Then the client
simply does the opposite on receipt.

Anyway, it seem to do the trick.  If anyone needs a copy of the source email me
and I will forward it.

Regards,
Colin Smith. 

A 100% Java zlib exists, and it's calle djzlib =)
http://www.jcraft.com/jzlib/

This a bug: The flush method does not work.

This problem can be fixed by:

- add the flush type argument to the deflate method
- implement the flush method: use Z_SYNC_FLUSH
- the deflate method really gets all data out of the 
deflater
- modify Deflater.java and deflater.c to support a flush 
type argument

These changes have been tested.

--- modified DeflaterStream.java ---

    public void write(byte[] b, int off, int len) throws 
IOException {
        if (def.finished()) {
            throw new IOException("write beyond end of 
stream");
        }
        if ((off | len | (off + len) | (b.length - (off + 
len))) < 0) {
            throw new IndexOutOfBoundsException();
        } else if (len == 0) {
            return;
        }
        if (!def.finished()) {
            def.setInput(b, off, len);
            deflate(Deflater.Z_NO_FLUSH);
        }
    }

    public void flush()
    throws IOException
    {
        if (!def.finished()) {
            deflate(Deflater.Z_SYNC_FLUSH);
        }
        out.flush();
    }
    
    protected void deflate(int aFlush) throws IOException
    {
        while (true) {
            int len = def.deflate(buf, 0, buf.length, 
aFlush);
            if (len > 0) {
                out.write(buf, 0, len);
            }
            if (def.needsInput() && len < buf.length) {
                break;
            }
        }
    }

--- original DeflaterStream.java ---

    public void write(byte[] b, int off, int len) throws 
IOException {
        if (def.finished()) {
            throw new IOException("write beyond end of 
stream");
        }
        if ((off | len | (off + len) | (b.length - (off + 
len))) < 0) {
            throw new IndexOutOfBoundsException();
        } else if (len == 0) {
            return;
        }
        if (!def.finished()) {
            def.setInput(b, off, len);
            while (!def.needsInput()) {
                deflate();
            }
        }
    }

    protected void deflate() throws IOException {
        int len = def.deflate(buf, 0, buf.length);
        if (len > 0) {
            out.write(buf, 0, len);
        }
    }


--- modified Deflater.java ---
    private native int deflateBytes(byte[] b, int off, int 
len, int aFlush);

--- original Deflater.java ---
    private native int deflateBytes(byte[] b, int off, int 
len);

--- modified deflater.c ---

            res = deflate(strm, finish ? Z_FINISH : 
aFlush);

Extending DeflaterOutputStream and overriding flush() as 
outlined below
yields a deflating OutputStream, that can be used for data 
transmission
(e.g. SocketOutputStream). 

Plase note, that InflaterInputStream should be subclassed 
as well in
order to prevent false blocking on reads. 

Furthermore (De/In)-flaters with nowrap == true should be 
used
to ommit headers/trailers.

rws@saddey.com 23-jun-02

The output part:

public class CompressingOutputStream extends 
DeflaterOutputStream {

    public CompressingOutputStream (final OutputStream out) 
{
      // Using Deflater with nowrap == true will ommit 
headers and trailers
      super(out, new Deflater
(Deflater.DEFAULT_COMPRESSION,true));
    }

    private static final byte [] EMPTYBYTEARRAY = new byte 
[0];
    /**
     * Insure all remaining data will be output.
     */
    public void flush() throws IOException {
        /**
         * Now this is tricky: We force the Deflater to 
flush
         * its data by switching compression level.
         * As yet, a perplexingly simple workaround for 
         * 
http://developer.java.sun.com/developer/bugParade/bugs/42557
43.html 
         */
        def.setInput(EMPTYBYTEARRAY, 0, 0);

        def.setLevel(Deflater.NO_COMPRESSION);
        deflate();

        def.setLevel(Deflater.DEFAULT_COMPRESSION);
        deflate();

        out.flush();
    }
} // class

The input part:

public class DecompressingInputStream extends 
InfalterInputStream {

    public DecompressingInputStream (final InputStream in) {
      // Using Inflater with nowrap == true will ommit 
headers and trailers
      super(in, new Inflater(true));
    }

    /**
     * available() should return the number of bytes that 
can be read without
     * running into blocking wait. Accomplishing this feast 
would eventually require
     * to pre-inflate a huge chunk of data, so we rather 
opt for a more relaxed
     * contract (java.util.zip.InflaterInputStream does not 
fit the bill). 
     * This code has been tested to work with 
BufferedReader.readLine();
     */
    public int available() throws IOException {
      if (!inf.finished() && !inf.needsInput()) {
        return 1;
      } else {
        return in.available();
      }
    }

} //class

The workaround suggested by jalapath works great except for
one small problem.  The code in the 'flush' method needs to
run only if 'def.finished()' returns false.  (e.g. if
(!def.finished()) { ... }

Important change, that I needed to make this work, it seems we are not done deflating until deflate doesn't yield a single byte:

	/*
	 * This has been changed to keep writing all data available from the deflater until
	 * it runs dry, using only needsInput() to see if we are done extracting data is b0rken!!!
	 */
	protected void deflate() throws IOException
	{
		int i;
		do {
	        i = def.deflate(buf, 0, buf.length);
	        if(i > 0) {
	            out.write(buf, 0, i);
	        }
		} while (i!=0); // Keep going until we start sucking mud
	}

This is not a "request for enhancement". This is a BUG
because the flush() method is BROKEN, because it fails
to "force any buffered output bytes to be written out".
Please recategorize as such.

What else can we do to persuade Sun that this is a serious deficiency in the Zip API in Java?

How many times must we run into this before someone decides to do the relatively trivial work to do a Z_SYNC_FLUSH on a call to flush()?

With the full Z_FINISH for each short message, I estimated that we were losing most of the benefits of compression (only getting about a 2:1 compression instead of a 6:1 or 8:1 or more that's typical of repetitive ASCII protocols).

Any chance this will make it into Mustang? It's a shame, this is from 1999...

Any chance of this happening before 3012? Come on guys, this is A BUG FROM 1999 WITH EIGHTYONE VOTES?

What else do you need? Pope's official request?

Happy Birthday!

You've reached the age of ten years, which is a truly exceptional achievement for a bug, especially one which would be so simple to fix.

Your enjoyable presence puts an element of surprise back into Java-based development, for instance discovering that a basic interface such as java.io.Flushable is misimplemented, and makes for a nice change from the otherwise boring reliability of the Java API.

So once again, I'd like to offer you my heartfelt congratulations for this outstanding feat. Keep going!

"For he is ja jolly good fellow..."

I give it ten more years until somebody recognize it as what it really is: A straightforward bug. Maybe I should round up my former project mates to vote for this so it gets a Top 25 "REF".

"...which nobody can deny!"

this is a ridiculous bug. flush doesn't comply w/ the general contract of 'flush' method of InputStream, basically it does nothing.
jzlib is a natural solution to such sickness, in the days of so good HotSpot it's no excuse to resort of C implementation for so a basic functionality.

Yes, Z_SYNC_FLUSH is all that's needed.  When will we get it?  This is really desperately needed.

10 years? Such a simple thing? That's what I don't really understand about Java... Anyway, I need it too for a protocol that uses XML with zlib compression. Looks like I have to use jzlib for the time being.