How to delete a line with variable search using sed command











up vote
2
down vote

favorite












What is wrong with the sed command on deleting the line which matches the input data?



InputData.txt



123,
1234,
1453,


Datatodelete.txt



1234,hellofirstline
123,hellosecondline
14676,hellothirdline
1453,hellofourthline


expected output in the Datatodelete.txt



14676,hellothirdline


Script:



echo "the script starts now"
while read EachLine
do
echo $EachLine
sed "/$EachLine/d" < /home/Datatodelete.txt >/home/dummy
done < /home/InputData.txt









share|improve this question




























    up vote
    2
    down vote

    favorite












    What is wrong with the sed command on deleting the line which matches the input data?



    InputData.txt



    123,
    1234,
    1453,


    Datatodelete.txt



    1234,hellofirstline
    123,hellosecondline
    14676,hellothirdline
    1453,hellofourthline


    expected output in the Datatodelete.txt



    14676,hellothirdline


    Script:



    echo "the script starts now"
    while read EachLine
    do
    echo $EachLine
    sed "/$EachLine/d" < /home/Datatodelete.txt >/home/dummy
    done < /home/InputData.txt









    share|improve this question


























      up vote
      2
      down vote

      favorite









      up vote
      2
      down vote

      favorite











      What is wrong with the sed command on deleting the line which matches the input data?



      InputData.txt



      123,
      1234,
      1453,


      Datatodelete.txt



      1234,hellofirstline
      123,hellosecondline
      14676,hellothirdline
      1453,hellofourthline


      expected output in the Datatodelete.txt



      14676,hellothirdline


      Script:



      echo "the script starts now"
      while read EachLine
      do
      echo $EachLine
      sed "/$EachLine/d" < /home/Datatodelete.txt >/home/dummy
      done < /home/InputData.txt









      share|improve this question















      What is wrong with the sed command on deleting the line which matches the input data?



      InputData.txt



      123,
      1234,
      1453,


      Datatodelete.txt



      1234,hellofirstline
      123,hellosecondline
      14676,hellothirdline
      1453,hellofourthline


      expected output in the Datatodelete.txt



      14676,hellothirdline


      Script:



      echo "the script starts now"
      while read EachLine
      do
      echo $EachLine
      sed "/$EachLine/d" < /home/Datatodelete.txt >/home/dummy
      done < /home/InputData.txt






      shell-script sed






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 25 at 14:46









      Rui F Ribeiro

      38.3k1475126




      38.3k1475126










      asked Mar 19 '14 at 15:06









      user3438085

      11112




      11112






















          3 Answers
          3






          active

          oldest

          votes

















          up vote
          5
          down vote













          Your sed command doesn't work because during the loop, each time it reads a line, it deletes that line (and only that line) from the full input file, outputting it to /home/dummy. This means that the output file gets overwritten each time. So the first iteration of the loop removes the line starting with 123, but then the second iteration uses the original full file which still includes this line.



          Try grep instead:



          grep -vFf /home/InputData.txt /home/Datatodelete.txt > /home/dummy


          From man grep:



             -F, --fixed-strings
          Interpret PATTERN as a list of fixed strings, separated by
          newlines, any of which is to be matched. (-F is specified by
          POSIX.)

          -f FILE, --file=FILE
          Obtain patterns from FILE, one per line. The empty file
          contains zero patterns, and therefore matches nothing. (-f is
          specified by POSIX.)
          -v, --invert-match
          Invert the sense of matching, to select non-matching lines. (-v
          is specified by POSIX.)





          share|improve this answer



















          • 2




            Depending on what else is in Input.txt you may want to use grep -F if there are characters that are intended to be matched verbatim. Eg .
            – Graeme
            Mar 19 '14 at 15:17










          • The above grep command is still not removing the records from Datatodelete.txt. It is still printing the whole content in dummy file
            – user3438085
            Mar 19 '14 at 21:00


















          up vote
          2
          down vote













          An awk version:



          awk -F, 'FNR==NR{a[$1];next} !($1 in a)' InputData.txt Datatodelete.txt





          share|improve this answer





















          • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
            – Flup
            Mar 19 '14 at 15:45










          • @Flup, it does provide the output the OP seeks.
            – glenn jackman
            Mar 19 '14 at 15:46










          • ...but doesn't answer the OP's question.
            – Flup
            Mar 19 '14 at 15:47






          • 1




            @Flup This isn't a story-telling competition.
            – devnull
            Mar 19 '14 at 16:22


















          up vote
          1
          down vote













          First, you don't want while read in there anywhere - sed will read your file. Next, you need to make sure that you handle greedy matches - sed will pull in as much as it can. So



              echo 1234 | sed -n '/123/p' 
          1234


          See? It prints it.



          So you need, based on what you've shown, something like this:



              </home/InputData.txt 
          sed -n '/1234/s//& hellofirstline/p;
          /123[^4]/s//& hellosecondline/p;
          /14676/s//& hellothirdline/p;
          /1453/s//& hellofourthline/p' >/home/dummy


          If your sed script is in a file:



              </home/InputData.txt 
          sed -nf ./delete.sed >/home/dummy





          share|improve this answer























            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "106"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














             

            draft saved


            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f120417%2fhow-to-delete-a-line-with-variable-search-using-sed-command%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            3 Answers
            3






            active

            oldest

            votes








            3 Answers
            3






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            5
            down vote













            Your sed command doesn't work because during the loop, each time it reads a line, it deletes that line (and only that line) from the full input file, outputting it to /home/dummy. This means that the output file gets overwritten each time. So the first iteration of the loop removes the line starting with 123, but then the second iteration uses the original full file which still includes this line.



            Try grep instead:



            grep -vFf /home/InputData.txt /home/Datatodelete.txt > /home/dummy


            From man grep:



               -F, --fixed-strings
            Interpret PATTERN as a list of fixed strings, separated by
            newlines, any of which is to be matched. (-F is specified by
            POSIX.)

            -f FILE, --file=FILE
            Obtain patterns from FILE, one per line. The empty file
            contains zero patterns, and therefore matches nothing. (-f is
            specified by POSIX.)
            -v, --invert-match
            Invert the sense of matching, to select non-matching lines. (-v
            is specified by POSIX.)





            share|improve this answer



















            • 2




              Depending on what else is in Input.txt you may want to use grep -F if there are characters that are intended to be matched verbatim. Eg .
              – Graeme
              Mar 19 '14 at 15:17










            • The above grep command is still not removing the records from Datatodelete.txt. It is still printing the whole content in dummy file
              – user3438085
              Mar 19 '14 at 21:00















            up vote
            5
            down vote













            Your sed command doesn't work because during the loop, each time it reads a line, it deletes that line (and only that line) from the full input file, outputting it to /home/dummy. This means that the output file gets overwritten each time. So the first iteration of the loop removes the line starting with 123, but then the second iteration uses the original full file which still includes this line.



            Try grep instead:



            grep -vFf /home/InputData.txt /home/Datatodelete.txt > /home/dummy


            From man grep:



               -F, --fixed-strings
            Interpret PATTERN as a list of fixed strings, separated by
            newlines, any of which is to be matched. (-F is specified by
            POSIX.)

            -f FILE, --file=FILE
            Obtain patterns from FILE, one per line. The empty file
            contains zero patterns, and therefore matches nothing. (-f is
            specified by POSIX.)
            -v, --invert-match
            Invert the sense of matching, to select non-matching lines. (-v
            is specified by POSIX.)





            share|improve this answer



















            • 2




              Depending on what else is in Input.txt you may want to use grep -F if there are characters that are intended to be matched verbatim. Eg .
              – Graeme
              Mar 19 '14 at 15:17










            • The above grep command is still not removing the records from Datatodelete.txt. It is still printing the whole content in dummy file
              – user3438085
              Mar 19 '14 at 21:00













            up vote
            5
            down vote










            up vote
            5
            down vote









            Your sed command doesn't work because during the loop, each time it reads a line, it deletes that line (and only that line) from the full input file, outputting it to /home/dummy. This means that the output file gets overwritten each time. So the first iteration of the loop removes the line starting with 123, but then the second iteration uses the original full file which still includes this line.



            Try grep instead:



            grep -vFf /home/InputData.txt /home/Datatodelete.txt > /home/dummy


            From man grep:



               -F, --fixed-strings
            Interpret PATTERN as a list of fixed strings, separated by
            newlines, any of which is to be matched. (-F is specified by
            POSIX.)

            -f FILE, --file=FILE
            Obtain patterns from FILE, one per line. The empty file
            contains zero patterns, and therefore matches nothing. (-f is
            specified by POSIX.)
            -v, --invert-match
            Invert the sense of matching, to select non-matching lines. (-v
            is specified by POSIX.)





            share|improve this answer














            Your sed command doesn't work because during the loop, each time it reads a line, it deletes that line (and only that line) from the full input file, outputting it to /home/dummy. This means that the output file gets overwritten each time. So the first iteration of the loop removes the line starting with 123, but then the second iteration uses the original full file which still includes this line.



            Try grep instead:



            grep -vFf /home/InputData.txt /home/Datatodelete.txt > /home/dummy


            From man grep:



               -F, --fixed-strings
            Interpret PATTERN as a list of fixed strings, separated by
            newlines, any of which is to be matched. (-F is specified by
            POSIX.)

            -f FILE, --file=FILE
            Obtain patterns from FILE, one per line. The empty file
            contains zero patterns, and therefore matches nothing. (-f is
            specified by POSIX.)
            -v, --invert-match
            Invert the sense of matching, to select non-matching lines. (-v
            is specified by POSIX.)






            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Mar 19 '14 at 15:33









            terdon

            126k31243419




            126k31243419










            answered Mar 19 '14 at 15:12









            Josh Jolly

            1,312812




            1,312812








            • 2




              Depending on what else is in Input.txt you may want to use grep -F if there are characters that are intended to be matched verbatim. Eg .
              – Graeme
              Mar 19 '14 at 15:17










            • The above grep command is still not removing the records from Datatodelete.txt. It is still printing the whole content in dummy file
              – user3438085
              Mar 19 '14 at 21:00














            • 2




              Depending on what else is in Input.txt you may want to use grep -F if there are characters that are intended to be matched verbatim. Eg .
              – Graeme
              Mar 19 '14 at 15:17










            • The above grep command is still not removing the records from Datatodelete.txt. It is still printing the whole content in dummy file
              – user3438085
              Mar 19 '14 at 21:00








            2




            2




            Depending on what else is in Input.txt you may want to use grep -F if there are characters that are intended to be matched verbatim. Eg .
            – Graeme
            Mar 19 '14 at 15:17




            Depending on what else is in Input.txt you may want to use grep -F if there are characters that are intended to be matched verbatim. Eg .
            – Graeme
            Mar 19 '14 at 15:17












            The above grep command is still not removing the records from Datatodelete.txt. It is still printing the whole content in dummy file
            – user3438085
            Mar 19 '14 at 21:00




            The above grep command is still not removing the records from Datatodelete.txt. It is still printing the whole content in dummy file
            – user3438085
            Mar 19 '14 at 21:00












            up vote
            2
            down vote













            An awk version:



            awk -F, 'FNR==NR{a[$1];next} !($1 in a)' InputData.txt Datatodelete.txt





            share|improve this answer





















            • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
              – Flup
              Mar 19 '14 at 15:45










            • @Flup, it does provide the output the OP seeks.
              – glenn jackman
              Mar 19 '14 at 15:46










            • ...but doesn't answer the OP's question.
              – Flup
              Mar 19 '14 at 15:47






            • 1




              @Flup This isn't a story-telling competition.
              – devnull
              Mar 19 '14 at 16:22















            up vote
            2
            down vote













            An awk version:



            awk -F, 'FNR==NR{a[$1];next} !($1 in a)' InputData.txt Datatodelete.txt





            share|improve this answer





















            • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
              – Flup
              Mar 19 '14 at 15:45










            • @Flup, it does provide the output the OP seeks.
              – glenn jackman
              Mar 19 '14 at 15:46










            • ...but doesn't answer the OP's question.
              – Flup
              Mar 19 '14 at 15:47






            • 1




              @Flup This isn't a story-telling competition.
              – devnull
              Mar 19 '14 at 16:22













            up vote
            2
            down vote










            up vote
            2
            down vote









            An awk version:



            awk -F, 'FNR==NR{a[$1];next} !($1 in a)' InputData.txt Datatodelete.txt





            share|improve this answer












            An awk version:



            awk -F, 'FNR==NR{a[$1];next} !($1 in a)' InputData.txt Datatodelete.txt






            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered Mar 19 '14 at 15:20









            cuonglm

            101k23196297




            101k23196297












            • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
              – Flup
              Mar 19 '14 at 15:45










            • @Flup, it does provide the output the OP seeks.
              – glenn jackman
              Mar 19 '14 at 15:46










            • ...but doesn't answer the OP's question.
              – Flup
              Mar 19 '14 at 15:47






            • 1




              @Flup This isn't a story-telling competition.
              – devnull
              Mar 19 '14 at 16:22


















            • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
              – Flup
              Mar 19 '14 at 15:45










            • @Flup, it does provide the output the OP seeks.
              – glenn jackman
              Mar 19 '14 at 15:46










            • ...but doesn't answer the OP's question.
              – Flup
              Mar 19 '14 at 15:47






            • 1




              @Flup This isn't a story-telling competition.
              – devnull
              Mar 19 '14 at 16:22
















            This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
            – Flup
            Mar 19 '14 at 15:45




            This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post.
            – Flup
            Mar 19 '14 at 15:45












            @Flup, it does provide the output the OP seeks.
            – glenn jackman
            Mar 19 '14 at 15:46




            @Flup, it does provide the output the OP seeks.
            – glenn jackman
            Mar 19 '14 at 15:46












            ...but doesn't answer the OP's question.
            – Flup
            Mar 19 '14 at 15:47




            ...but doesn't answer the OP's question.
            – Flup
            Mar 19 '14 at 15:47




            1




            1




            @Flup This isn't a story-telling competition.
            – devnull
            Mar 19 '14 at 16:22




            @Flup This isn't a story-telling competition.
            – devnull
            Mar 19 '14 at 16:22










            up vote
            1
            down vote













            First, you don't want while read in there anywhere - sed will read your file. Next, you need to make sure that you handle greedy matches - sed will pull in as much as it can. So



                echo 1234 | sed -n '/123/p' 
            1234


            See? It prints it.



            So you need, based on what you've shown, something like this:



                </home/InputData.txt 
            sed -n '/1234/s//& hellofirstline/p;
            /123[^4]/s//& hellosecondline/p;
            /14676/s//& hellothirdline/p;
            /1453/s//& hellofourthline/p' >/home/dummy


            If your sed script is in a file:



                </home/InputData.txt 
            sed -nf ./delete.sed >/home/dummy





            share|improve this answer



























              up vote
              1
              down vote













              First, you don't want while read in there anywhere - sed will read your file. Next, you need to make sure that you handle greedy matches - sed will pull in as much as it can. So



                  echo 1234 | sed -n '/123/p' 
              1234


              See? It prints it.



              So you need, based on what you've shown, something like this:



                  </home/InputData.txt 
              sed -n '/1234/s//& hellofirstline/p;
              /123[^4]/s//& hellosecondline/p;
              /14676/s//& hellothirdline/p;
              /1453/s//& hellofourthline/p' >/home/dummy


              If your sed script is in a file:



                  </home/InputData.txt 
              sed -nf ./delete.sed >/home/dummy





              share|improve this answer

























                up vote
                1
                down vote










                up vote
                1
                down vote









                First, you don't want while read in there anywhere - sed will read your file. Next, you need to make sure that you handle greedy matches - sed will pull in as much as it can. So



                    echo 1234 | sed -n '/123/p' 
                1234


                See? It prints it.



                So you need, based on what you've shown, something like this:



                    </home/InputData.txt 
                sed -n '/1234/s//& hellofirstline/p;
                /123[^4]/s//& hellosecondline/p;
                /14676/s//& hellothirdline/p;
                /1453/s//& hellofourthline/p' >/home/dummy


                If your sed script is in a file:



                    </home/InputData.txt 
                sed -nf ./delete.sed >/home/dummy





                share|improve this answer














                First, you don't want while read in there anywhere - sed will read your file. Next, you need to make sure that you handle greedy matches - sed will pull in as much as it can. So



                    echo 1234 | sed -n '/123/p' 
                1234


                See? It prints it.



                So you need, based on what you've shown, something like this:



                    </home/InputData.txt 
                sed -n '/1234/s//& hellofirstline/p;
                /123[^4]/s//& hellosecondline/p;
                /14676/s//& hellothirdline/p;
                /1453/s//& hellofourthline/p' >/home/dummy


                If your sed script is in a file:



                    </home/InputData.txt 
                sed -nf ./delete.sed >/home/dummy






                share|improve this answer














                share|improve this answer



                share|improve this answer








                edited Mar 19 '14 at 15:20

























                answered Mar 19 '14 at 15:14









                mikeserv

                45.1k566152




                45.1k566152






























                     

                    draft saved


                    draft discarded



















































                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f120417%2fhow-to-delete-a-line-with-variable-search-using-sed-command%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Accessing regular linux commands in Huawei's Dopra Linux

                    Can't connect RFCOMM socket: Host is down

                    Kernel panic - not syncing: Fatal Exception in Interrupt