Sstrhsrtj

File,

TABLE1  

-------  

1234



TABLE1  

-------  

9555  



TABLE1  

-------  

87676  



TABLE1

-------  

2344

I want the output like

TABLE1  

-------  

1234

9555  

87676

2344

Here is one liner, using sed and awk :

sed '/^$/d' filename | awk '!a[$1]++' 

Combination of grep and awk :

grep . filename | awk '!a[$1]++'

As @cas suggested, You can do that in single awk command also.

awk '!x[$1]++ && ! /^[[:blank:]]*$/' filename

You can use awk '!x[$1]++' file > file_new

While trying this command, I was getting one extra new line in the file you have given.

I modified this to awk '!x[$1]++' file | sed '/^$/d' > file_new, which should solve your problem for this case.

I usually use sort and uniq together to get rid of duplicates like this:

    cat file | sort | uniq

However, with your input, it will end up like this:

    -------

    1234

    2344

    87676

    9555

    TABLE1

This command removes all but the numbers and adds the header afterwards:

    cat file | grep '^[[:digit:]]*$'| grep -v '^$' |sed '1iTABLE1n-------'

and gives you this result:

    TABLE1

    -------

    1234

    9555

    87676

    2344

Use the command uniq, you can remove duplicate entries. Like :

cat file | sort -r | uniq

But in this specific case is not producing exactly the expected result as the file must be sorted for uniq to work - it will only detect duplicate lines if they are adjacent.

Another solution would be to read file and skip the lines containing TABLE or ---- (except first occurrence) :

count_t=0

count_d=0

while read line; do 

  if [[ $line == "TABLE"* ]] ; then  

    if [[ $count_t -eq 0 ]]; then 

      ((count_t++)) 

    else

      continue 

    fi 

  fi

  if [[ $line == "-----"* ]] ; then  

    if [[ $count_d -eq 0 ]]; then 

      ((count_d++)) 

    else

      continue 

    fi

  fi

  echo $line

done < file

The awk and sed solutions posted by others are better though.

Even though this is an old thread, I would like to contribute this answer that uses only a single sed command:

sed '1,2p;/^[[:digit:]]/!d;' file

It keeps the two first lines (the heading and underline), then deletes every line that doesn't start with a digit.