Originally posted by Satrajit Ghosh:
Indeed, I agree with Satra that as data gets bigger and computing power increases faster than bandwidth, storing huge data to disk is going to become more and more of a bottleneck. In other words, I/O on large data files will be very problematic soon.

Two methods are usually employed to mitigate these problems. First making the data access local on the disk. This requires abandonning the hyper-cube representation for storage and saving the data as blocks, or chunks, that can be sliced locally on the disk in every direction.
Second, compression, if it is implemented with a seekable and stream-capable compression algorithm (not gzip), can render the I/O faster. Both approaches can be combinned.

Optimal implementation of the above suggestion is a lot of work. However, the HDF library has tackled these problems efficiently, and it already available in most scientific computing environment. As a matter of fact, even Mathwork, a company with huge resources, ditched their own I/O library to use HDF5 for the most recent data saving formats.