Remove all duplicate word from string using shell script











up vote
7
down vote

favorite
1












I have a string like 



"aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"


I want to remove duplicate word from string then output will be like 



"aaa,bbb,ccc"


I tried This code Source



$ echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


It is working fine with same value,but when I give my variable value then it is showing all duplicate word also.



How can I remove duplicate value.



UPDATE



My question is adding all corresponding value into a single string if user is same .I have data like this ->



   user name    | colour

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green


In coding I fetch all distinct user then I concatenate color string successfully .For that I am using code -



while read the records 



    if [ "$c" == "" ]; then  #$c I defined global

        c="$colour1"

    else

        c="$c,$colour1" 

    fi


When I print this $c variable i get the output (For User AAA) 



"red,black,blue,red,green,red,black,blue,red,green,"


I want to remove duplicate color .Then desired output should be like



"red,black,blue,green"


For this desired output i used above code 



 echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


but it is displaying the output with duplicate values .Like



"red,black,blue,red,green,red,black,blue,red,green,"
Thanks






shell-script shell text-processing xargs duplicate







share|improve this question




















  • 3




    Please clarify what is wrong with what you are using. I don't understand what you mean by "when I give my variable value". What value do you give? Where does it fail?
    – terdon♦
    Mar 23 '17 at 12:57










  • echo 'aaa aaa aaa bbb bbb ccc bbb ccc' | xargs -n1 | sort -u | xargs gives aaa bbb ccc.. so you need to show exact code you tired and output you got.. with the string in variable: s='aaa aaa aaa bbb bbb ccc bbb ccc'; echo "$s" | xargs -n1 | sort -u | xargs
    – Sundeep
    Mar 23 '17 at 13:01










  • string value comes dynamically. It is printing same value (contain duplicate value).
    – Urvashi
    Mar 23 '17 at 13:02






  • 1




    yeah, show the code that failed, otherwise how would we know what could've gone wrong?
    – Sundeep
    Mar 23 '17 at 13:02










  • Does the order matter?
    – Jacob Vlijm
    Mar 23 '17 at 14:06















up vote
7
down vote

favorite
1












I have a string like 



"aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"


I want to remove duplicate word from string then output will be like 



"aaa,bbb,ccc"


I tried This code Source



$ echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


It is working fine with same value,but when I give my variable value then it is showing all duplicate word also.



How can I remove duplicate value.



UPDATE



My question is adding all corresponding value into a single string if user is same .I have data like this ->



   user name    | colour

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green


In coding I fetch all distinct user then I concatenate color string successfully .For that I am using code -



while read the records 



    if [ "$c" == "" ]; then  #$c I defined global

        c="$colour1"

    else

        c="$c,$colour1" 

    fi


When I print this $c variable i get the output (For User AAA) 



"red,black,blue,red,green,red,black,blue,red,green,"


I want to remove duplicate color .Then desired output should be like



"red,black,blue,green"


For this desired output i used above code 



 echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


but it is displaying the output with duplicate values .Like



"red,black,blue,red,green,red,black,blue,red,green,"
Thanks






shell-script shell text-processing xargs duplicate







share|improve this question




















  • 3




    Please clarify what is wrong with what you are using. I don't understand what you mean by "when I give my variable value". What value do you give? Where does it fail?
    – terdon♦
    Mar 23 '17 at 12:57










  • echo 'aaa aaa aaa bbb bbb ccc bbb ccc' | xargs -n1 | sort -u | xargs gives aaa bbb ccc.. so you need to show exact code you tired and output you got.. with the string in variable: s='aaa aaa aaa bbb bbb ccc bbb ccc'; echo "$s" | xargs -n1 | sort -u | xargs
    – Sundeep
    Mar 23 '17 at 13:01










  • string value comes dynamically. It is printing same value (contain duplicate value).
    – Urvashi
    Mar 23 '17 at 13:02






  • 1




    yeah, show the code that failed, otherwise how would we know what could've gone wrong?
    – Sundeep
    Mar 23 '17 at 13:02










  • Does the order matter?
    – Jacob Vlijm
    Mar 23 '17 at 14:06













up vote
7
down vote

favorite
1









up vote
7
down vote

favorite
1






1





I have a string like 



"aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"


I want to remove duplicate word from string then output will be like 



"aaa,bbb,ccc"


I tried This code Source



$ echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


It is working fine with same value,but when I give my variable value then it is showing all duplicate word also.



How can I remove duplicate value.



UPDATE



My question is adding all corresponding value into a single string if user is same .I have data like this ->



   user name    | colour

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green


In coding I fetch all distinct user then I concatenate color string successfully .For that I am using code -



while read the records 



    if [ "$c" == "" ]; then  #$c I defined global

        c="$colour1"

    else

        c="$c,$colour1" 

    fi


When I print this $c variable i get the output (For User AAA) 



"red,black,blue,red,green,red,black,blue,red,green,"


I want to remove duplicate color .Then desired output should be like



"red,black,blue,green"


For this desired output i used above code 



 echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


but it is displaying the output with duplicate values .Like



"red,black,blue,red,green,red,black,blue,red,green,"
Thanks






shell-script shell text-processing xargs duplicate







share|improve this question















I have a string like 



"aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"


I want to remove duplicate word from string then output will be like 



"aaa,bbb,ccc"


I tried This code Source



$ echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


It is working fine with same value,but when I give my variable value then it is showing all duplicate word also.



How can I remove duplicate value.



UPDATE



My question is adding all corresponding value into a single string if user is same .I have data like this ->



   user name    | colour

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green

    AAA         | red

    AAA         | black

    BBB         | red

    BBB         | blue

    AAA         | blue

    AAA         | red

    CCC         | red

    CCC         | red

    AAA         | green


In coding I fetch all distinct user then I concatenate color string successfully .For that I am using code -



while read the records 



    if [ "$c" == "" ]; then  #$c I defined global

        c="$colour1"

    else

        c="$c,$colour1" 

    fi


When I print this $c variable i get the output (For User AAA) 



"red,black,blue,red,green,red,black,blue,red,green,"


I want to remove duplicate color .Then desired output should be like



"red,black,blue,green"


For this desired output i used above code 



 echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs


but it is displaying the output with duplicate values .Like



"red,black,blue,red,green,red,black,blue,red,green,"
Thanks






shell-script shell text-processing xargs duplicate




shell-script shell text-processing xargs duplicate






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited May 23 '17 at 12:39









Community♦

1




1










asked Mar 23 '17 at 12:41









Urvashi

7316




7316








  • 3




    Please clarify what is wrong with what you are using. I don't understand what you mean by "when I give my variable value". What value do you give? Where does it fail?
    – terdon♦
    Mar 23 '17 at 12:57










  • echo 'aaa aaa aaa bbb bbb ccc bbb ccc' | xargs -n1 | sort -u | xargs gives aaa bbb ccc.. so you need to show exact code you tired and output you got.. with the string in variable: s='aaa aaa aaa bbb bbb ccc bbb ccc'; echo "$s" | xargs -n1 | sort -u | xargs
    – Sundeep
    Mar 23 '17 at 13:01










  • string value comes dynamically. It is printing same value (contain duplicate value).
    – Urvashi
    Mar 23 '17 at 13:02






  • 1




    yeah, show the code that failed, otherwise how would we know what could've gone wrong?
    – Sundeep
    Mar 23 '17 at 13:02










  • Does the order matter?
    – Jacob Vlijm
    Mar 23 '17 at 14:06














  • 3




    Please clarify what is wrong with what you are using. I don't understand what you mean by "when I give my variable value". What value do you give? Where does it fail?
    – terdon♦
    Mar 23 '17 at 12:57










  • echo 'aaa aaa aaa bbb bbb ccc bbb ccc' | xargs -n1 | sort -u | xargs gives aaa bbb ccc.. so you need to show exact code you tired and output you got.. with the string in variable: s='aaa aaa aaa bbb bbb ccc bbb ccc'; echo "$s" | xargs -n1 | sort -u | xargs
    – Sundeep
    Mar 23 '17 at 13:01










  • string value comes dynamically. It is printing same value (contain duplicate value).
    – Urvashi
    Mar 23 '17 at 13:02






  • 1




    yeah, show the code that failed, otherwise how would we know what could've gone wrong?
    – Sundeep
    Mar 23 '17 at 13:02










  • Does the order matter?
    – Jacob Vlijm
    Mar 23 '17 at 14:06








3




3




Please clarify what is wrong with what you are using. I don't understand what you mean by "when I give my variable value". What value do you give? Where does it fail?
– terdon♦
Mar 23 '17 at 12:57




Please clarify what is wrong with what you are using. I don't understand what you mean by "when I give my variable value". What value do you give? Where does it fail?
– terdon♦
Mar 23 '17 at 12:57












echo 'aaa aaa aaa bbb bbb ccc bbb ccc' | xargs -n1 | sort -u | xargs gives aaa bbb ccc.. so you need to show exact code you tired and output you got.. with the string in variable: s='aaa aaa aaa bbb bbb ccc bbb ccc'; echo "$s" | xargs -n1 | sort -u | xargs
– Sundeep
Mar 23 '17 at 13:01




echo 'aaa aaa aaa bbb bbb ccc bbb ccc' | xargs -n1 | sort -u | xargs gives aaa bbb ccc.. so you need to show exact code you tired and output you got.. with the string in variable: s='aaa aaa aaa bbb bbb ccc bbb ccc'; echo "$s" | xargs -n1 | sort -u | xargs
– Sundeep
Mar 23 '17 at 13:01












string value comes dynamically. It is printing same value (contain duplicate value).
– Urvashi
Mar 23 '17 at 13:02




string value comes dynamically. It is printing same value (contain duplicate value).
– Urvashi
Mar 23 '17 at 13:02




1




1




yeah, show the code that failed, otherwise how would we know what could've gone wrong?
– Sundeep
Mar 23 '17 at 13:02




yeah, show the code that failed, otherwise how would we know what could've gone wrong?
– Sundeep
Mar 23 '17 at 13:02












Does the order matter?
– Jacob Vlijm
Mar 23 '17 at 14:06




Does the order matter?
– Jacob Vlijm
Mar 23 '17 at 14:06










9 Answers
9






active

oldest

votes

















up vote
7
down vote



accepted










One more awk, just for fun:



$ a="aaa bbb aaa bbb ccc aaa ddd bbb ccc"

$ echo "$a" | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("n")}'

aaa bbb ccc ddd


By the way, even your solution works fine with variables:



$ b="zebra ant spider spider ant zebra ant" 

$ echo "$b" | xargs -n1 | sort -u | xargs

ant spider zebra





share|improve this answer























  • This works for me .Thanks @George Vasiliou
    – Urvashi
    Mar 24 '17 at 5:59


















up vote
8
down vote













$ echo "zebra ant spider spider ant zebra ant"  | awk -v RS="[ n]+" '!n[$0]++' 

zebra

ant

spider





share|improve this answer

















  • 1




    Very clever!!!!
    – George Vasiliou
    Mar 24 '17 at 0:54










  • @GeorgeVasiliou, thank you [or to tell the truth, very lazy :-) ]
    – JJoao
    Mar 24 '17 at 8:44


















up vote
6
down vote













With tr, sort and uniq



echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq


or 



echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq | xargs


to get one line






share|improve this answer























  • You need to add | xargs to join the output to one line again
    – Philippos
    Mar 23 '17 at 12:59






  • 3




    Or use sort -u. Or even a awk '!u[$0]++.
    – Benoît
    Mar 23 '17 at 18:42






  • 1




    @Benoît Wow, I did not know about sort -u. I've been using sort | uniq all this time. The wasted keystrokes...
    – gardenhead
    Mar 24 '17 at 1:25


















up vote
2
down vote













With gnu sed:



sed ':s;s/(<S*>)(.*)<1>/12/g;ts'


You may add ;s/ */ /g to remove dublicate spaces.



Functions like this: If a word is a second time in this line, remove it and start over until no dublication is found anymore.






share|improve this answer





















  • What are < and >?
    – someonewithpc
    Mar 23 '17 at 20:19










  • @someonewithpc They match no character, but the beginning and end of a word to prevent substrings from being matched.
    – Philippos
    Mar 23 '17 at 21:29










  • Nice, but is that portable? Also, aren't words separated by whitespace? Seems redundant to match not whitespace followed by the end of a word.
    – someonewithpc
    Mar 23 '17 at 21:34






  • 1




    @someonewithpc No, it's not standard, that's why I wrote gnu sed. The nice part is that you don't have to handle first and last string separately
    – Philippos
    Mar 23 '17 at 21:44


















up vote
2
down vote













perl -lane '$,=$";print grep { ! $h{$_}++ } @F'





share|improve this answer




























    up vote
    2
    down vote













    Obligatory awk solution:



    $ echo "ant zebra ant spider spider ant zebra ant" | 
    
   awk -vRS=" " -vORS=" " '!a[$1] {a[$1]++} END{ for (x in a) print x;  } ' ; echo
    
zebra ant spider


    (The final echo is there for the newline)






    share|improve this answer























    • Plus one for the awk ! I was builting also an awk solution just for fun. There is a slight possibility words to be printed in random order at END section due to the random way that awk itterates in array keys.
      – George Vasiliou
      Mar 23 '17 at 14:14










    • Yes, they will be printed in an essentially random order. The sort solution doesn't keep the original order either, though.
      – ilkkachu
      Mar 23 '17 at 14:17










    • Yes, good point! Even sort prints in different order than input.
      – George Vasiliou
      Mar 23 '17 at 14:18






    • 1




      @ilkkachu Actually we don't need to wait for the input to end. We can make decision to print or not to print with a slight modification to your code: awk -vRS=" " -vORS=" " '!a[$1]++ {print $1}' ; echo This preserves the order.
      – user218374
      Mar 23 '17 at 14:31




















    up vote
    1
    down vote













    Python



    Option 1





    #!/usr/bin/env python
    
# get_unique_words.py
    

    
import sys
    

    
l = 
    
for w in sys.argv[1].split(','):
    
  if w not in l:
    
    l += [ w ]
    
print ','.join(l)


    Make executable, then call from Bash:



    $ ./get_unique_words.py "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"
    
aaa,bbb,ccc


    Or you could implement it as a Bash function, but the syntax is messy.



    get_unique_words(){
    
  python -c "
    
l = 
    
for w in '$1'.split(','):
    
  if w not in l:
    
    l += [ w ]
    
print ','.join(l)"
    
}


    Option 2



    This option can become a one-liner if needed:





    #!/usr/bin/env python
    
# get_unique_words.py
    

    
import sys
    

    
s_in = sys.argv[1]
    
l_in = s_in.split(',') # Turn string into a list.
    
set_out = set(l_in) # Turning a list into a set removes duplicates items.
    
s_out = ','.join(set_out) 
    
print s_out


    In Bash:



    get_unique_words(){
    
  python -c "print ','.join(set('$1'.split(',')))"
    
}





    share|improve this answer






























      up vote
      0
      down vote













      cat filename | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }' > newfile






      share|improve this answer








      New contributor




      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.


















      • I do not get it
        – Pierre.Vriens
        Dec 2 at 7:00


















      up vote
      -2
      down vote













      a="aaa aaa aaa bbb bbb ccc bbb ccc"
      
for item in $a
      
do
      
   echo $item
      
done | sort -u | (while read i; do ans="$ans $i"; done ; echo $ans)





      share|improve this answer























      • Please add an explanation on how your code works and why you did this and that.
        – xhienne
        Mar 24 '17 at 1:37











      Your Answer








      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "106"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      convertImagesToLinks: false,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: null,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f353321%2fremove-all-duplicate-word-from-string-using-shell-script%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      9 Answers
      9






      active

      oldest

      votes








      9 Answers
      9






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes








      up vote
      7
      down vote



      accepted










      One more awk, just for fun:



      $ a="aaa bbb aaa bbb ccc aaa ddd bbb ccc"
      
$ echo "$a" | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("n")}'
      
aaa bbb ccc ddd


      By the way, even your solution works fine with variables:



      $ b="zebra ant spider spider ant zebra ant" 
      
$ echo "$b" | xargs -n1 | sort -u | xargs
      
ant spider zebra





      share|improve this answer























      • This works for me .Thanks @George Vasiliou
        – Urvashi
        Mar 24 '17 at 5:59















      up vote
      7
      down vote



      accepted










      One more awk, just for fun:



      $ a="aaa bbb aaa bbb ccc aaa ddd bbb ccc"
      
$ echo "$a" | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("n")}'
      
aaa bbb ccc ddd


      By the way, even your solution works fine with variables:



      $ b="zebra ant spider spider ant zebra ant" 
      
$ echo "$b" | xargs -n1 | sort -u | xargs
      
ant spider zebra





      share|improve this answer























      • This works for me .Thanks @George Vasiliou
        – Urvashi
        Mar 24 '17 at 5:59













      up vote
      7
      down vote



      accepted







      up vote
      7
      down vote



      accepted






      One more awk, just for fun:



      $ a="aaa bbb aaa bbb ccc aaa ddd bbb ccc"
      
$ echo "$a" | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("n")}'
      
aaa bbb ccc ddd


      By the way, even your solution works fine with variables:



      $ b="zebra ant spider spider ant zebra ant" 
      
$ echo "$b" | xargs -n1 | sort -u | xargs
      
ant spider zebra





      share|improve this answer














      One more awk, just for fun:



      $ a="aaa bbb aaa bbb ccc aaa ddd bbb ccc"
      
$ echo "$a" | awk '{for (i=1;i<=NF;i++) if (!a[$i]++) printf("%s%s",$i,FS)}{printf("n")}'
      
aaa bbb ccc ddd


      By the way, even your solution works fine with variables:



      $ b="zebra ant spider spider ant zebra ant" 
      
$ echo "$b" | xargs -n1 | sort -u | xargs
      
ant spider zebra






      share|improve this answer














      share|improve this answer



      share|improve this answer








      edited Mar 23 '17 at 14:20

























      answered Mar 23 '17 at 14:12









      George Vasiliou

      5,57531028




      5,57531028












      • This works for me .Thanks @George Vasiliou
        – Urvashi
        Mar 24 '17 at 5:59


















      • This works for me .Thanks @George Vasiliou
        – Urvashi
        Mar 24 '17 at 5:59
















      This works for me .Thanks @George Vasiliou
      – Urvashi
      Mar 24 '17 at 5:59




      This works for me .Thanks @George Vasiliou
      – Urvashi
      Mar 24 '17 at 5:59












      up vote
      8
      down vote













      $ echo "zebra ant spider spider ant zebra ant"  | awk -v RS="[ n]+" '!n[$0]++' 
      
zebra
      
ant
      
spider





      share|improve this answer

















      • 1




        Very clever!!!!
        – George Vasiliou
        Mar 24 '17 at 0:54










      • @GeorgeVasiliou, thank you [or to tell the truth, very lazy :-) ]
        – JJoao
        Mar 24 '17 at 8:44















      up vote
      8
      down vote













      $ echo "zebra ant spider spider ant zebra ant"  | awk -v RS="[ n]+" '!n[$0]++' 
      
zebra
      
ant
      
spider





      share|improve this answer

















      • 1




        Very clever!!!!
        – George Vasiliou
        Mar 24 '17 at 0:54










      • @GeorgeVasiliou, thank you [or to tell the truth, very lazy :-) ]
        – JJoao
        Mar 24 '17 at 8:44













      up vote
      8
      down vote










      up vote
      8
      down vote









      $ echo "zebra ant spider spider ant zebra ant"  | awk -v RS="[ n]+" '!n[$0]++' 
      
zebra
      
ant
      
spider





      share|improve this answer












      $ echo "zebra ant spider spider ant zebra ant"  | awk -v RS="[ n]+" '!n[$0]++' 
      
zebra
      
ant
      
spider






      share|improve this answer












      share|improve this answer



      share|improve this answer










      answered Mar 23 '17 at 15:25









      JJoao

      7,0041827




      7,0041827








      • 1




        Very clever!!!!
        – George Vasiliou
        Mar 24 '17 at 0:54










      • @GeorgeVasiliou, thank you [or to tell the truth, very lazy :-) ]
        – JJoao
        Mar 24 '17 at 8:44














      • 1




        Very clever!!!!
        – George Vasiliou
        Mar 24 '17 at 0:54










      • @GeorgeVasiliou, thank you [or to tell the truth, very lazy :-) ]
        – JJoao
        Mar 24 '17 at 8:44








      1




      1




      Very clever!!!!
      – George Vasiliou
      Mar 24 '17 at 0:54




      Very clever!!!!
      – George Vasiliou
      Mar 24 '17 at 0:54












      @GeorgeVasiliou, thank you [or to tell the truth, very lazy :-) ]
      – JJoao
      Mar 24 '17 at 8:44




      @GeorgeVasiliou, thank you [or to tell the truth, very lazy :-) ]
      – JJoao
      Mar 24 '17 at 8:44










      up vote
      6
      down vote













      With tr, sort and uniq



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq


      or 



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq | xargs


      to get one line






      share|improve this answer























      • You need to add | xargs to join the output to one line again
        – Philippos
        Mar 23 '17 at 12:59






      • 3




        Or use sort -u. Or even a awk '!u[$0]++.
        – Benoît
        Mar 23 '17 at 18:42






      • 1




        @Benoît Wow, I did not know about sort -u. I've been using sort | uniq all this time. The wasted keystrokes...
        – gardenhead
        Mar 24 '17 at 1:25















      up vote
      6
      down vote













      With tr, sort and uniq



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq


      or 



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq | xargs


      to get one line






      share|improve this answer























      • You need to add | xargs to join the output to one line again
        – Philippos
        Mar 23 '17 at 12:59






      • 3




        Or use sort -u. Or even a awk '!u[$0]++.
        – Benoît
        Mar 23 '17 at 18:42






      • 1




        @Benoît Wow, I did not know about sort -u. I've been using sort | uniq all this time. The wasted keystrokes...
        – gardenhead
        Mar 24 '17 at 1:25













      up vote
      6
      down vote










      up vote
      6
      down vote









      With tr, sort and uniq



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq


      or 



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq | xargs


      to get one line






      share|improve this answer














      With tr, sort and uniq



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq


      or 



      echo "zebra ant spider spider ant zebra ant" | tr ' ' 'n' | sort | uniq | xargs


      to get one line







      share|improve this answer














      share|improve this answer



      share|improve this answer








      edited Mar 23 '17 at 13:01

























      answered Mar 23 '17 at 12:55









      Michael D.

      1,534816




      1,534816












      • You need to add | xargs to join the output to one line again
        – Philippos
        Mar 23 '17 at 12:59






      • 3




        Or use sort -u. Or even a awk '!u[$0]++.
        – Benoît
        Mar 23 '17 at 18:42






      • 1




        @Benoît Wow, I did not know about sort -u. I've been using sort | uniq all this time. The wasted keystrokes...
        – gardenhead
        Mar 24 '17 at 1:25


















      • You need to add | xargs to join the output to one line again
        – Philippos
        Mar 23 '17 at 12:59






      • 3




        Or use sort -u. Or even a awk '!u[$0]++.
        – Benoît
        Mar 23 '17 at 18:42






      • 1




        @Benoît Wow, I did not know about sort -u. I've been using sort | uniq all this time. The wasted keystrokes...
        – gardenhead
        Mar 24 '17 at 1:25
















      You need to add | xargs to join the output to one line again
      – Philippos
      Mar 23 '17 at 12:59




      You need to add | xargs to join the output to one line again
      – Philippos
      Mar 23 '17 at 12:59




      3




      3




      Or use sort -u. Or even a awk '!u[$0]++.
      – Benoît
      Mar 23 '17 at 18:42




      Or use sort -u. Or even a awk '!u[$0]++.
      – Benoît
      Mar 23 '17 at 18:42




      1




      1




      @Benoît Wow, I did not know about sort -u. I've been using sort | uniq all this time. The wasted keystrokes...
      – gardenhead
      Mar 24 '17 at 1:25




      @Benoît Wow, I did not know about sort -u. I've been using sort | uniq all this time. The wasted keystrokes...
      – gardenhead
      Mar 24 '17 at 1:25










      up vote
      2
      down vote













      With gnu sed:



      sed ':s;s/(<S*>)(.*)<1>/12/g;ts'


      You may add ;s/ */ /g to remove dublicate spaces.



      Functions like this: If a word is a second time in this line, remove it and start over until no dublication is found anymore.






      share|improve this answer





















      • What are < and >?
        – someonewithpc
        Mar 23 '17 at 20:19










      • @someonewithpc They match no character, but the beginning and end of a word to prevent substrings from being matched.
        – Philippos
        Mar 23 '17 at 21:29










      • Nice, but is that portable? Also, aren't words separated by whitespace? Seems redundant to match not whitespace followed by the end of a word.
        – someonewithpc
        Mar 23 '17 at 21:34






      • 1




        @someonewithpc No, it's not standard, that's why I wrote gnu sed. The nice part is that you don't have to handle first and last string separately
        – Philippos
        Mar 23 '17 at 21:44















      up vote
      2
      down vote













      With gnu sed:



      sed ':s;s/(<S*>)(.*)<1>/12/g;ts'


      You may add ;s/ */ /g to remove dublicate spaces.



      Functions like this: If a word is a second time in this line, remove it and start over until no dublication is found anymore.






      share|improve this answer





















      • What are < and >?
        – someonewithpc
        Mar 23 '17 at 20:19










      • @someonewithpc They match no character, but the beginning and end of a word to prevent substrings from being matched.
        – Philippos
        Mar 23 '17 at 21:29










      • Nice, but is that portable? Also, aren't words separated by whitespace? Seems redundant to match not whitespace followed by the end of a word.
        – someonewithpc
        Mar 23 '17 at 21:34






      • 1




        @someonewithpc No, it's not standard, that's why I wrote gnu sed. The nice part is that you don't have to handle first and last string separately
        – Philippos
        Mar 23 '17 at 21:44













      up vote
      2
      down vote










      up vote
      2
      down vote









      With gnu sed:



      sed ':s;s/(<S*>)(.*)<1>/12/g;ts'


      You may add ;s/ */ /g to remove dublicate spaces.



      Functions like this: If a word is a second time in this line, remove it and start over until no dublication is found anymore.






      share|improve this answer












      With gnu sed:



      sed ':s;s/(<S*>)(.*)<1>/12/g;ts'


      You may add ;s/ */ /g to remove dublicate spaces.



      Functions like this: If a word is a second time in this line, remove it and start over until no dublication is found anymore.







      share|improve this answer












      share|improve this answer



      share|improve this answer










      answered Mar 23 '17 at 12:52









      Philippos

      5,98211547




      5,98211547












      • What are < and >?
        – someonewithpc
        Mar 23 '17 at 20:19










      • @someonewithpc They match no character, but the beginning and end of a word to prevent substrings from being matched.
        – Philippos
        Mar 23 '17 at 21:29










      • Nice, but is that portable? Also, aren't words separated by whitespace? Seems redundant to match not whitespace followed by the end of a word.
        – someonewithpc
        Mar 23 '17 at 21:34






      • 1




        @someonewithpc No, it's not standard, that's why I wrote gnu sed. The nice part is that you don't have to handle first and last string separately
        – Philippos
        Mar 23 '17 at 21:44


















      • What are < and >?
        – someonewithpc
        Mar 23 '17 at 20:19










      • @someonewithpc They match no character, but the beginning and end of a word to prevent substrings from being matched.
        – Philippos
        Mar 23 '17 at 21:29










      • Nice, but is that portable? Also, aren't words separated by whitespace? Seems redundant to match not whitespace followed by the end of a word.
        – someonewithpc
        Mar 23 '17 at 21:34






      • 1




        @someonewithpc No, it's not standard, that's why I wrote gnu sed. The nice part is that you don't have to handle first and last string separately
        – Philippos
        Mar 23 '17 at 21:44
















      What are < and >?
      – someonewithpc
      Mar 23 '17 at 20:19




      What are < and >?
      – someonewithpc
      Mar 23 '17 at 20:19












      @someonewithpc They match no character, but the beginning and end of a word to prevent substrings from being matched.
      – Philippos
      Mar 23 '17 at 21:29




      @someonewithpc They match no character, but the beginning and end of a word to prevent substrings from being matched.
      – Philippos
      Mar 23 '17 at 21:29












      Nice, but is that portable? Also, aren't words separated by whitespace? Seems redundant to match not whitespace followed by the end of a word.
      – someonewithpc
      Mar 23 '17 at 21:34




      Nice, but is that portable? Also, aren't words separated by whitespace? Seems redundant to match not whitespace followed by the end of a word.
      – someonewithpc
      Mar 23 '17 at 21:34




      1




      1




      @someonewithpc No, it's not standard, that's why I wrote gnu sed. The nice part is that you don't have to handle first and last string separately
      – Philippos
      Mar 23 '17 at 21:44




      @someonewithpc No, it's not standard, that's why I wrote gnu sed. The nice part is that you don't have to handle first and last string separately
      – Philippos
      Mar 23 '17 at 21:44










      up vote
      2
      down vote













      perl -lane '$,=$";print grep { ! $h{$_}++ } @F'





      share|improve this answer

























        up vote
        2
        down vote













        perl -lane '$,=$";print grep { ! $h{$_}++ } @F'





        share|improve this answer























          up vote
          2
          down vote










          up vote
          2
          down vote









          perl -lane '$,=$";print grep { ! $h{$_}++ } @F'





          share|improve this answer












          perl -lane '$,=$";print grep { ! $h{$_}++ } @F'






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Mar 23 '17 at 13:07







          user218374





























              up vote
              2
              down vote













              Obligatory awk solution:



              $ echo "ant zebra ant spider spider ant zebra ant" | 
              
   awk -vRS=" " -vORS=" " '!a[$1] {a[$1]++} END{ for (x in a) print x;  } ' ; echo
              
zebra ant spider


              (The final echo is there for the newline)






              share|improve this answer























              • Plus one for the awk ! I was builting also an awk solution just for fun. There is a slight possibility words to be printed in random order at END section due to the random way that awk itterates in array keys.
                – George Vasiliou
                Mar 23 '17 at 14:14










              • Yes, they will be printed in an essentially random order. The sort solution doesn't keep the original order either, though.
                – ilkkachu
                Mar 23 '17 at 14:17










              • Yes, good point! Even sort prints in different order than input.
                – George Vasiliou
                Mar 23 '17 at 14:18






              • 1




                @ilkkachu Actually we don't need to wait for the input to end. We can make decision to print or not to print with a slight modification to your code: awk -vRS=" " -vORS=" " '!a[$1]++ {print $1}' ; echo This preserves the order.
                – user218374
                Mar 23 '17 at 14:31

















              up vote
              2
              down vote













              Obligatory awk solution:



              $ echo "ant zebra ant spider spider ant zebra ant" | 
              
   awk -vRS=" " -vORS=" " '!a[$1] {a[$1]++} END{ for (x in a) print x;  } ' ; echo
              
zebra ant spider


              (The final echo is there for the newline)






              share|improve this answer























              • Plus one for the awk ! I was builting also an awk solution just for fun. There is a slight possibility words to be printed in random order at END section due to the random way that awk itterates in array keys.
                – George Vasiliou
                Mar 23 '17 at 14:14










              • Yes, they will be printed in an essentially random order. The sort solution doesn't keep the original order either, though.
                – ilkkachu
                Mar 23 '17 at 14:17










              • Yes, good point! Even sort prints in different order than input.
                – George Vasiliou
                Mar 23 '17 at 14:18






              • 1




                @ilkkachu Actually we don't need to wait for the input to end. We can make decision to print or not to print with a slight modification to your code: awk -vRS=" " -vORS=" " '!a[$1]++ {print $1}' ; echo This preserves the order.
                – user218374
                Mar 23 '17 at 14:31















              up vote
              2
              down vote










              up vote
              2
              down vote









              Obligatory awk solution:



              $ echo "ant zebra ant spider spider ant zebra ant" | 
              
   awk -vRS=" " -vORS=" " '!a[$1] {a[$1]++} END{ for (x in a) print x;  } ' ; echo
              
zebra ant spider


              (The final echo is there for the newline)






              share|improve this answer














              Obligatory awk solution:



              $ echo "ant zebra ant spider spider ant zebra ant" | 
              
   awk -vRS=" " -vORS=" " '!a[$1] {a[$1]++} END{ for (x in a) print x;  } ' ; echo
              
zebra ant spider


              (The final echo is there for the newline)







              share|improve this answer














              share|improve this answer



              share|improve this answer








              edited Mar 23 '17 at 13:58

























              answered Mar 23 '17 at 13:52









              ilkkachu

              54.2k782147




              54.2k782147












              • Plus one for the awk ! I was builting also an awk solution just for fun. There is a slight possibility words to be printed in random order at END section due to the random way that awk itterates in array keys.
                – George Vasiliou
                Mar 23 '17 at 14:14










              • Yes, they will be printed in an essentially random order. The sort solution doesn't keep the original order either, though.
                – ilkkachu
                Mar 23 '17 at 14:17










              • Yes, good point! Even sort prints in different order than input.
                – George Vasiliou
                Mar 23 '17 at 14:18






              • 1




                @ilkkachu Actually we don't need to wait for the input to end. We can make decision to print or not to print with a slight modification to your code: awk -vRS=" " -vORS=" " '!a[$1]++ {print $1}' ; echo This preserves the order.
                – user218374
                Mar 23 '17 at 14:31




















              • Plus one for the awk ! I was builting also an awk solution just for fun. There is a slight possibility words to be printed in random order at END section due to the random way that awk itterates in array keys.
                – George Vasiliou
                Mar 23 '17 at 14:14










              • Yes, they will be printed in an essentially random order. The sort solution doesn't keep the original order either, though.
                – ilkkachu
                Mar 23 '17 at 14:17










              • Yes, good point! Even sort prints in different order than input.
                – George Vasiliou
                Mar 23 '17 at 14:18






              • 1




                @ilkkachu Actually we don't need to wait for the input to end. We can make decision to print or not to print with a slight modification to your code: awk -vRS=" " -vORS=" " '!a[$1]++ {print $1}' ; echo This preserves the order.
                – user218374
                Mar 23 '17 at 14:31


















              Plus one for the awk ! I was builting also an awk solution just for fun. There is a slight possibility words to be printed in random order at END section due to the random way that awk itterates in array keys.
              – George Vasiliou
              Mar 23 '17 at 14:14




              Plus one for the awk ! I was builting also an awk solution just for fun. There is a slight possibility words to be printed in random order at END section due to the random way that awk itterates in array keys.
              – George Vasiliou
              Mar 23 '17 at 14:14












              Yes, they will be printed in an essentially random order. The sort solution doesn't keep the original order either, though.
              – ilkkachu
              Mar 23 '17 at 14:17




              Yes, they will be printed in an essentially random order. The sort solution doesn't keep the original order either, though.
              – ilkkachu
              Mar 23 '17 at 14:17












              Yes, good point! Even sort prints in different order than input.
              – George Vasiliou
              Mar 23 '17 at 14:18




              Yes, good point! Even sort prints in different order than input.
              – George Vasiliou
              Mar 23 '17 at 14:18




              1




              1




              @ilkkachu Actually we don't need to wait for the input to end. We can make decision to print or not to print with a slight modification to your code: awk -vRS=" " -vORS=" " '!a[$1]++ {print $1}' ; echo This preserves the order.
              – user218374
              Mar 23 '17 at 14:31






              @ilkkachu Actually we don't need to wait for the input to end. We can make decision to print or not to print with a slight modification to your code: awk -vRS=" " -vORS=" " '!a[$1]++ {print $1}' ; echo This preserves the order.
              – user218374
              Mar 23 '17 at 14:31












              up vote
              1
              down vote













              Python



              Option 1





              #!/usr/bin/env python
              
# get_unique_words.py
              

              
import sys
              

              
l = 
              
for w in sys.argv[1].split(','):
              
  if w not in l:
              
    l += [ w ]
              
print ','.join(l)


              Make executable, then call from Bash:



              $ ./get_unique_words.py "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"
              
aaa,bbb,ccc


              Or you could implement it as a Bash function, but the syntax is messy.



              get_unique_words(){
              
  python -c "
              
l = 
              
for w in '$1'.split(','):
              
  if w not in l:
              
    l += [ w ]
              
print ','.join(l)"
              
}


              Option 2



              This option can become a one-liner if needed:





              #!/usr/bin/env python
              
# get_unique_words.py
              

              
import sys
              

              
s_in = sys.argv[1]
              
l_in = s_in.split(',') # Turn string into a list.
              
set_out = set(l_in) # Turning a list into a set removes duplicates items.
              
s_out = ','.join(set_out) 
              
print s_out


              In Bash:



              get_unique_words(){
              
  python -c "print ','.join(set('$1'.split(',')))"
              
}





              share|improve this answer



























                up vote
                1
                down vote













                Python



                Option 1





                #!/usr/bin/env python
                
# get_unique_words.py
                

                
import sys
                

                
l = 
                
for w in sys.argv[1].split(','):
                
  if w not in l:
                
    l += [ w ]
                
print ','.join(l)


                Make executable, then call from Bash:



                $ ./get_unique_words.py "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"
                
aaa,bbb,ccc


                Or you could implement it as a Bash function, but the syntax is messy.



                get_unique_words(){
                
  python -c "
                
l = 
                
for w in '$1'.split(','):
                
  if w not in l:
                
    l += [ w ]
                
print ','.join(l)"
                
}


                Option 2



                This option can become a one-liner if needed:





                #!/usr/bin/env python
                
# get_unique_words.py
                

                
import sys
                

                
s_in = sys.argv[1]
                
l_in = s_in.split(',') # Turn string into a list.
                
set_out = set(l_in) # Turning a list into a set removes duplicates items.
                
s_out = ','.join(set_out) 
                
print s_out


                In Bash:



                get_unique_words(){
                
  python -c "print ','.join(set('$1'.split(',')))"
                
}





                share|improve this answer

























                  up vote
                  1
                  down vote










                  up vote
                  1
                  down vote









                  Python



                  Option 1





                  #!/usr/bin/env python
                  
# get_unique_words.py
                  

                  
import sys
                  

                  
l = 
                  
for w in sys.argv[1].split(','):
                  
  if w not in l:
                  
    l += [ w ]
                  
print ','.join(l)


                  Make executable, then call from Bash:



                  $ ./get_unique_words.py "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"
                  
aaa,bbb,ccc


                  Or you could implement it as a Bash function, but the syntax is messy.



                  get_unique_words(){
                  
  python -c "
                  
l = 
                  
for w in '$1'.split(','):
                  
  if w not in l:
                  
    l += [ w ]
                  
print ','.join(l)"
                  
}


                  Option 2



                  This option can become a one-liner if needed:





                  #!/usr/bin/env python
                  
# get_unique_words.py
                  

                  
import sys
                  

                  
s_in = sys.argv[1]
                  
l_in = s_in.split(',') # Turn string into a list.
                  
set_out = set(l_in) # Turning a list into a set removes duplicates items.
                  
s_out = ','.join(set_out) 
                  
print s_out


                  In Bash:



                  get_unique_words(){
                  
  python -c "print ','.join(set('$1'.split(',')))"
                  
}





                  share|improve this answer














                  Python



                  Option 1





                  #!/usr/bin/env python
                  
# get_unique_words.py
                  

                  
import sys
                  

                  
l = 
                  
for w in sys.argv[1].split(','):
                  
  if w not in l:
                  
    l += [ w ]
                  
print ','.join(l)


                  Make executable, then call from Bash:



                  $ ./get_unique_words.py "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc"
                  
aaa,bbb,ccc


                  Or you could implement it as a Bash function, but the syntax is messy.



                  get_unique_words(){
                  
  python -c "
                  
l = 
                  
for w in '$1'.split(','):
                  
  if w not in l:
                  
    l += [ w ]
                  
print ','.join(l)"
                  
}


                  Option 2



                  This option can become a one-liner if needed:





                  #!/usr/bin/env python
                  
# get_unique_words.py
                  

                  
import sys
                  

                  
s_in = sys.argv[1]
                  
l_in = s_in.split(',') # Turn string into a list.
                  
set_out = set(l_in) # Turning a list into a set removes duplicates items.
                  
s_out = ','.join(set_out) 
                  
print s_out


                  In Bash:



                  get_unique_words(){
                  
  python -c "print ','.join(set('$1'.split(',')))"
                  
}






                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited May 14 '17 at 3:19

























                  answered Mar 23 '17 at 20:34









                  wjandrea

                  466413




                  466413






















                      up vote
                      0
                      down vote













                      cat filename | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }' > newfile






                      share|improve this answer








                      New contributor




                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.


















                      • I do not get it
                        – Pierre.Vriens
                        Dec 2 at 7:00















                      up vote
                      0
                      down vote













                      cat filename | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }' > newfile






                      share|improve this answer








                      New contributor




                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.


















                      • I do not get it
                        – Pierre.Vriens
                        Dec 2 at 7:00













                      up vote
                      0
                      down vote










                      up vote
                      0
                      down vote









                      cat filename | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }' > newfile






                      share|improve this answer








                      New contributor




                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.









                      cat filename | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }' > newfile







                      share|improve this answer








                      New contributor




                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.









                      share|improve this answer



                      share|improve this answer






                      New contributor




                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.









                      answered Dec 2 at 4:18









                      天津神 こと

                      1




                      1




                      New contributor




                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.





                      New contributor





                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.






                      天津神 こと is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                      Check out our Code of Conduct.












                      • I do not get it
                        – Pierre.Vriens
                        Dec 2 at 7:00


















                      • I do not get it
                        – Pierre.Vriens
                        Dec 2 at 7:00
















                      I do not get it
                      – Pierre.Vriens
                      Dec 2 at 7:00




                      I do not get it
                      – Pierre.Vriens
                      Dec 2 at 7:00










                      up vote
                      -2
                      down vote













                      a="aaa aaa aaa bbb bbb ccc bbb ccc"
                      
for item in $a
                      
do
                      
   echo $item
                      
done | sort -u | (while read i; do ans="$ans $i"; done ; echo $ans)





                      share|improve this answer























                      • Please add an explanation on how your code works and why you did this and that.
                        – xhienne
                        Mar 24 '17 at 1:37















                      up vote
                      -2
                      down vote













                      a="aaa aaa aaa bbb bbb ccc bbb ccc"
                      
for item in $a
                      
do
                      
   echo $item
                      
done | sort -u | (while read i; do ans="$ans $i"; done ; echo $ans)





                      share|improve this answer























                      • Please add an explanation on how your code works and why you did this and that.
                        – xhienne
                        Mar 24 '17 at 1:37













                      up vote
                      -2
                      down vote










                      up vote
                      -2
                      down vote









                      a="aaa aaa aaa bbb bbb ccc bbb ccc"
                      
for item in $a
                      
do
                      
   echo $item
                      
done | sort -u | (while read i; do ans="$ans $i"; done ; echo $ans)





                      share|improve this answer














                      a="aaa aaa aaa bbb bbb ccc bbb ccc"
                      
for item in $a
                      
do
                      
   echo $item
                      
done | sort -u | (while read i; do ans="$ans $i"; done ; echo $ans)






                      share|improve this answer














                      share|improve this answer



                      share|improve this answer








                      edited Mar 24 '17 at 0:27

























                      answered Mar 24 '17 at 0:18









                      Tododo Fly

                      11




                      11












                      • Please add an explanation on how your code works and why you did this and that.
                        – xhienne
                        Mar 24 '17 at 1:37


















                      • Please add an explanation on how your code works and why you did this and that.
                        – xhienne
                        Mar 24 '17 at 1:37
















                      Please add an explanation on how your code works and why you did this and that.
                      – xhienne
                      Mar 24 '17 at 1:37




                      Please add an explanation on how your code works and why you did this and that.
                      – xhienne
                      Mar 24 '17 at 1:37


















                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Unix & Linux Stack Exchange!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.





                      Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                      Please pay close attention to the following guidance:


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f353321%2fremove-all-duplicate-word-from-string-using-shell-script%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      -

                      Popular posts from this blog

                      サソリ

                      Image
                      この記事は検証可能な参考文献や出典が全く示されていないか、不十分です。 出典を追加して記事の信頼性向上にご協力ください。 （ 2019年1月 ） 「 サソリ 」のその他の用法については「サソリ (曖昧さ回避)」をご覧ください。 「 スコーピオン 」と「 スコルピオン 」はこの項目へ転送されています。その他の用法については「スコーピオン」をご覧ください。 サソリ目 生息年代: 430–0 Ma PreЄ Є O S D C P T J K Pg N シルル紀前期-現世 ダイオウサソリ（ Pandinus Imperator ） 分類 界 : 動物界 Animalia 門 : 節足動物門 Arthropoda 亜門 : 鋏角亜門 Chelicerata 綱 : クモガタ綱 Arachnida 目 : サソリ目 Scorpiones 学名 Scorpiones C. L. Koch, 1837 英名 Scorpion 上科 Pseudochactoidea キョクトウサソリ上科 Buthoidea Chaeriloidea Chactoidea Iuroidea Scorpionoidea サソリ （ サソリ類 、 蠍 、 蝎 、 scorpion ）は、鋏角亜門・クモガタ綱の サソリ目 （ Scorpiones ）に属する節足動物の総称である。体の前端に鋏型の触肢、後端に毒針を有する捕食者である。 1700種以上を含め、最古の化石記録は4億3千万年前のシルル紀まで遡る。有毒生物として広く知られるが、人間の命にかかわる毒を持つものはごく一部の種類に過ぎない。 目次 1 形態 1.1 前体 1.2 後体 1.2.1 中体 1.2.2 終体 2 雌雄 3 生態 3.1 天敵 3.2 生殖 4 分布 5 毒 6 蛍光現象 7 分類
                      Read more

                      広島県道265号伴広島線

                      Image
                      一般県道 広島県道265号伴広島線 路線延長 km 起点 広島県広島市安佐南区伴東7丁目 終点 広島県広島市中区榎町 接続する 主な道路 （記法） 広島県道38号広島豊平線 広島県道71号広島湯来線 山陽自動車道五日市IC 国道183号（ 国道261号重複） ■テンプレート（■ノート ■使い方） ■PJ道路 広島県道265号伴広島線 （ひろしまけんどう265ごう ともひろしません）は広島県広島市安佐南区から広島県広島市中区を結ぶ一般県道である。 目次 1 路線データ 2 通過する自治体 3 見どころ 4 接続道路 4.1 本線 4.2 旧道 5 別名 6 関連項目 路線データ 本線 起点：広島市安佐南区伴東7丁目・大原下橋南詰交差点（広島県道38号広島豊平線（本線・旧道）交点） 終点：広島市中区榎町・十日市交差点（国道183号（国道261号重用）交点） 総延長： 旧道 起点：広島市安佐南区伴東4丁目・新細坂交差点（広島県道38号広島豊平線交点） 終点：広島市安佐南区伴中央4丁目・寺組橋(南)交差点（広島県道265号伴広島線（本線）交点） 総延長： 通過する自治体 広島県 広島市（安佐南区、佐伯区、西区、中区） 見どころ 広島高速交通広島新交通1号線（アストラムライン） 広島広域公園 広島広域公園陸上競技場（広島ビッグアーチ） 己斐峠 己斐橋（太田川放水路） 天満橋（天満川） 接続道路 本線 広島県道38号広島豊平線（本線・旧道）（広島市安佐南区伴東7丁目・大原下橋南詰交差点〔起点〕） 広島県道71号広島湯来線（本線）（広島市安佐南区伴中央2丁目・伴交番入口交差点 - 広島市安佐南区伴中央4丁目・寺組橋(南)交差点間で重用） 広島県道71号広島湯来線（旧道）（広島県道265号伴広島線（旧道）重用）（広島市安佐南区伴中央4丁目・寺組橋(南)交差点） 広島県道71号広島湯来線（広島市安佐南区伴中央4丁目・寺組橋(南)交差点 - 広島市佐伯区五日市町石内・五月が丘(南)交差点間で重用）
                      Read more

                      Setup Asymptote in Texstudio

                      Image
                      0 Good time of the day! I am trying to compile the following file: documentclass[12pt]{article} usepackage[inline]{asymptote} begin{document} Here is a venn diagram produced with Asymptote, drawn to width 4cm: defA{A} defB{V{B}} begin{figure} begin{center} begin{asy} size(4cm,0); pen colour1=red; pen colour2=green; pair z0=(0,0); pair z1=(-1,0); pair z2=(1,0); real r=1.5; path c1=circle(z1,r); path c2=circle(z2,r); fill(c1,colour1); fill(c2,colour2); picture intersection=new picture; fill(intersection,c1,colour1+colour2); clip(intersection,c2); add(intersection); draw(c1); draw(c2); //draw("$A$",box,z1)
                      Read more