File Compression methods

Levi Pearson levipearson at
Thu Oct 10 11:20:52 MDT 2013

On Thu, Oct 10, 2013 at 10:53 AM, Rich <rich at> wrote:
> On Thu, Oct 10, 2013 at 10:48:09AM -0600, Rich wrote:
>> On Thu, Oct 10, 2013 at 01:55:03PM +0530, Dan Egli wrote:
>>> And a two
>>> step process is unfortunately out of the question. The machines will only
>>> have either 750GB or 1TB hdds, which obviously won't work for extracting
>>> the tar to disk then extracting from the tar on disk. tar's extraction
>>> process would run out of space before it finished.
> Whoops, I misunderstood what you meant, forget what I said about that
> (except that it's still true, it just doesn't address your concern).

Because tar and gzip can both take input from stdin and write output
to stdout, you can compose them in such a way that they become,
effectively, a single step. And then you can compose them with
rsh/ssh/etc. in order to eliminate the entire intermediate file. So,
the composition of an archiving codec, a compression codec, and a
remote shell process is *effectively* a single-step image transfer
system, at least as long as you choose codecs that are capable of
operating in a chunked manner rather than requiring random access or
the entire input/output at once.

This idea is one of the pillars of the UNIX programming philosophy,
and also is given a more general and rigorous treatment in the basis
of functional programming.

More information about the PLUG mailing list