all versions to date

The File.list() method returns String[].  File.list(FilenameFilter) also returns
String[].  The problem is that when a directory has tens of thousands of files
in it, the use of File.list() can cause a huge VM size explosion.  In most
applications processing files, it is not necessary to see all of the files at
once.  One file at a time is fine.  The Posix readdir functionality in
particular is handy in these cases so that one can process all the entries
in a directory, one at a time.

I'd like to suggest that a new method be provided in File that will facilitate
the convienent processing of large directories.  Here's my suggestion.

public void scan( FileEntryAction act ) throws IOException;

public interface FileEntryAction {
    pubic void processFile( File dir, String name );
}

This would allow the activities to be implemented in another class if needed
and eliminate the need for such a huge array for large directories.

The reason I am purposing this instead of the alteration of the behavior of
File.list( FilenameFilter ) to filter as files are found, is that I think the
above purposal is more general solution and avoids the array creation
altogether.
(Review ID: 97165) 
======================================================================
Posted Date : 2005-08-25 16:31:25.0

There is no work around.  For large directories, java can spends minutes if not
hours scanning a directory to collect all the entries, and not only that, but it
causes the VM size to swell excessively in the process.
======================================================================

Incremental reads of large directories will be addressed in the new I/O
framework.  --   xxxxx@xxxxx   1999/10/28
Contribution forum : https://jdk-collaboration.dev.java.net/servlets/ProjectForumMessageView?forumID=1463&messageID=23037
Posted Date : 2008-02-01 05:46:19.0

This issue has been addressed by the file system API defined for NIO2. A DirectoryStream is returned when the directory is opened and this can be used to iterate over the entries in a huge directory without requiring the read the entire list of file names as is done by the File.list method.
Posted Date : 2008-02-01 10:48:49.0

In the FileEntryAction interface, processFile should throw IOException.  I'd
like to see more
information on what the "new I/O framework" entails.  Where can I
find more information?

When will this new I/O frame work be available for use in JDK1.1?

gregg:  The "New I/O Framework" is in java.nio.* in 1.4 
beta.  I doubt that these classes will be accessible from 
1.1 code, at least not in the most general case.

The interface proposed above would work nicely as an 
addition to the API, as it is quite flexible... though it 
would be nice to allow it to throw IOExceptions.  A less 
general solution would be to provide a list() style 
function that returned an Iterator.

I check the New I/O framework in 1.4.  I do not see how 
this is addressed.

I don't see that this is addressed in the New I/O routines at all.  Why can't we subclass the FileSystem 
provider for the platform so that File.list could just return the first 'n' files when it is called or something.  I 
really need a solution to this problem!

What is the state of this issue?  Can we expect something 
in 1.4.1 or 1.5?  I need to know so that I can stupe to 
stupid JNI tricks if there will be no work on this.

In this case it means in the new filesystem API being worked 
on for Tiger (1.5).
The RFE number is 4313887 and it is in JSR 203:
http://jcp.org/en/jsr/detail?id=203
Mike McCloskey
Scheduled for 1st quarter 2004... 

Sorry, I think this RFE is addressed with Microsoft .NET. 
Thanks Sun, there is no need to address this RFE anymore - 
more than four years are more than enough - or nearly seven 
(7!!!!) years for RFE 4057701.

is this issue handled with nio.2 / 1.6? or 1.7, 1.8, 1.9...? 
ever/never?! who knows...

1999: "Incremental reads of large directories will be addressed in the new I/O framework."

Let me guess:
"This RFE is part of JSR-203 (see http://www.jcp.org/en/home/index), which is targeted for J2SE 1.6"?!

And the shifted again to J2SE 1.7 (aka 7.0)?

first there will be men on mars  (or sun is sold to m$) before sun will resolve a five year old rfe.

Yes!!! Now it

Yes!!!

Now it's targetted for 1.7 (dolphin)...

Let me guess... 2006/2007 it is targetted for 1.8?!

Look at me:
http://forums.java.net/jive/thread.jspa?threadID=454&tstart=0

How about :
     java.util.Enumeration File.listFiles();

This implementation of the enumerator  instead of reading and loading the info of all files at once would just load it on demand.

Returning an Enumeration might have been the preferred solution, oh, eight years ago or so.  Since 1.2 came out an Iterator would be preferred - and since 5.0 came out, an Iterable<File> would be the way to go.  The embedded Iterator<File> would be able to return some results as soon as they were available (without requiring the whole File[] in memory) and by using an Iterable rather than Iterator, the result can be used in an enhanced for.  (General advice for post-Tiger code: prefer Iterable to either Iterator or Collection, unless you actualy need to do something besides iterate through the result.)

Of course, the exact object returned is just window dressing at this point.  The key point is that Sun would have to actually *act* on this RFE sometime.  Grumble, grumble...

[quote]
Evaluation  	

Incremental reads of large directories will be addressed in the new I/O
framework.  --  xxxxx@xxxxx  1999/10/28
[/quote]

Come on... Is the "new I/O framework" beeing defined?

http://forums.java.net/jive/thread.jspa?threadID=454&tstart=0

I need readdir() style functionality in Java.  Please add this feature because loading an entire list of 100,000+ files is silly.