Large sequential I/O can cause performance problems due to excessive use of the memory page cache. One way to avoid this problem is to use direct I/O on filesystems where large sequential I/Os are common. Direct I/O is a mechanism for bypassing the memory page cache alltogether.
DIRECTIO_ON allows you to force directio per-file and is used by Oracle database. The idea is that with a properly sized memory area the database can manage its own buffering and avoid the overhead of copying to kernel buffers. Even after the database is shut down directio will persist for a long time. The flag is set in the inode’s i_flag and hangs around cached within the inode in the DNLC and will effect files until the inode is flushed from the DNLC.
John Alderson demonstrated the problem using two simple C programs.