Sstrhsrtj

I want to have an xargs-like interface to operate transparently on large tar balls without unpacking the whole archive at once. I've made already this shell script prototype xargs-tar which unpacks a tar ball into RAM-disk (/dev/shm), processes all files when they appear, and deletes all processed files immediately.

Here it is, xargs-tar:

#!/bin/bash



TAR_FILE="$1"

shift



TMP_ROOT="/dev/shm" # ...or /tmp

TMP_DIR="$(mktemp -d "$TMP_ROOT/xargs-tar-XXXXXX")"

UNTAR_DIR="$TMP_DIR/untar"

FILE_LIST="$TMP_DIR/files-list"

EXEC_FILE="$TMP_DIR/exec-file"



mkdir -p "$UNTAR_DIR"

mkfifo "$FILE_LIST"



(

    # single quotes for user's command and args

    for i in "$@"; do

            echo -n "'$i' "

    done

    echo '"$@"'

    echo 'rm "$@"'

) > "$EXEC_FILE"

chmod u+x "$EXEC_FILE"



# Background untar. Write file list (zero terminated, no directories) to named

# pipe. The output is one line delayed to make sure we print only finished file

# names.

(

    tar -v -C "$UNTAR_DIR" -xf "$TAR_FILE" 

            | awk 'BEGIN {last = "";}

                   !//$/ {if (last != "") print last; last=$0;}

                   END {if (last != "") print last;}' 

            | tr 'n' ''

) > "$FILE_LIST" &



cd "$UNTAR_DIR"

xargs --null -r sh "$EXEC_FILE" < "$FILE_LIST"



rm -rf "$TMP_DIR"

Example usage:

./xargs-tar  palemoon.tar.bz2  wc -l

      546 palemoon/libsoftokn3.so

     1437 palemoon/libnss3.so

[...]

      267 palemoon/libnssutil3.so

      220 palemoon/libsmime3.so

        6 palemoon/defaults/pref/channel-prefs.js

   379727 total

which should be equivalent (but faster and less disk-usage) to:

tar -xf palemoon.tar.bz2

find palemoon -type f -print0 | xargs -0 wc -l

rm -rf palemoon

Of course my xargs-tar prototype needs a lot of improvements, like

• interface to limit max temp space (regarding available memory)


• pausing untar when the consumer is too slow


• error handling (duplicate files, whatever ...)


• supporting other archive formats, not only tar


• etc.

That's why I'm thinking about starting a serious project, implementation in C.

My question is now: Does something like this already exist? Would other people find it useful too? Do I waste my time?

I know tarfs, really useful but not exactly what I want. I want fast, simple piped command lines, portable implementation. The key is: The un-tarred files are processed while they are still cached, and then deleted immediately.