Sstrhsrtj

I have a data file with 10,000 columns and 117,000 rows. my original data has a lot of repetition within each columns. it is like:

inputfile.txt :

    123 124 111 

    321 124 111 

    123 000 111 

    123 111 222

I want to keep one copy of each value within each column like:

    123 124 111

    321 000 222

        111 

I need a program to do all columns together since I have 10,000 columns.

This should do what you require in 5 lines of code (2 of which are just tidying):

#!/bin/bash

# run this, specifying input file as $1 (parameter 1)



# delete any pre-existing column files from /tmp

find /tmp -maxdepth 1 -name "column*" -delete



# create /tmp/columnN files - each file holds one column of $1

awk '{for (f=1; f<=NF; f++) {print $f >>"/tmp/column"f}}' "$1"



# iterate through column files, sorting and removing duplicates

find /tmp -maxdepth 1 -name "column*" -execdir sort -o {} -u {} ;



# re-combine columns and output to stdout

paste /tmp/column*



# delete column files from /tmp

find /tmp -maxdepth 1 -name "column*" -delete

It is possible that with a very large number of columns (as you have) the paste command will fail because /tmp/column* cannot be fully expanded.

A difference in output from your example is that each column's output is sorted whereas in your original the 2nd column was unsorted.