FILE_FLAG_SEQUENTIAL_FTW

I’ve been toying with file system optimization recently as we’re starting to implement a background loading system. At pdimov’s advice, I tried setting FILE_FLAG_SEQUENTIAL_SCAN to all Windows file open operations, and got an amazing 10% of absolutely automatic reduction in load time… an excellent case of money left on the table. (The load process has already been optimized in several iterations, so shaving off 10% by another means would not be trivial.)

It’s of great educational value to examine all kinds of logs and statistics produced by the file system. For example, I estimated that more than 70% of high-level file read operations would consume an entire file; the real number turned out to be around 10%. I found all sorts of weird leftover tiny reads, like skeletons being read one bone at a time, or scripts being loaded via a 512 byte buffer. We probably have a wristwatch programmer embedded undercover in our team…

The very existence of functions like fread() is some kind of lie-to-children, like the fact that the earth is round (it is not) or that the atom is a little planet. People who pride themselves of writing in a “to-the-metal” language like C or C++ to squeeze maximal performance out of the hardware, and still use fread (or, heaven forbid, iostreams), should take some time to read about low-level esoterica like FILE_FLAG_NO_BUFFERED with its sector-aligned reads bypassing the system cache, IO completion ports for file access, or even the scatter/gather API used to build your own cache. This thesis by Jan Wassenberg  quotes 10x improvement in effective reading throughput for a real-world loading sequence over the plain read().

One Response to “FILE_FLAG_SEQUENTIAL_FTW”

  1. Cheap Ugg Boots Says:

    I love to read this type of stuff. Good and attractive information I take from it..Thank you for posting such a nice article

Leave a Reply