Converting data and using sed/awk
up vote
-1
down vote
favorite
I have a file which is provided below, Col1/Field 1 will always have duplicate data the rest of the fields may/may not but am not worried of other columns, all I need is if there is duplicate data it need to be printed with empty space for col1 and the values of Field 1 will be ordered so no values repeats further down the rows.
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
A1 | 98 | P
A1 | 98 | P
B1 | 98 | P
B1 | 98 | P
B1 | 98 | P
C1 | 98 | P
C1 | 98 | P
C1 | 98 | P
need to convert and the awk/sed need to be applied on col1 only
(the output should look like below after awk/sed/cut is used)
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
awk '!x[$1]++' file <-- is removing whole line
awk/sed/cut anything is fine.
Wrote 50 lines of code , struck at last phase
shell
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
asked Nov 28 at 3:38
vj56789
61
add a comment |
up vote
-1
down vote
favorite
I have a file which is provided below, Col1/Field 1 will always have duplicate data the rest of the fields may/may not but am not worried of other columns, all I need is if there is duplicate data it need to be printed with empty space for col1 and the values of Field 1 will be ordered so no values repeats further down the rows.
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
A1 | 98 | P
A1 | 98 | P
B1 | 98 | P
B1 | 98 | P
B1 | 98 | P
C1 | 98 | P
C1 | 98 | P
C1 | 98 | P
need to convert and the awk/sed need to be applied on col1 only
(the output should look like below after awk/sed/cut is used)
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
awk '!x[$1]++' file <-- is removing whole line
awk/sed/cut anything is fine.
Wrote 50 lines of code , struck at last phase
shell
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
asked Nov 28 at 3:38
vj56789
61
add a comment |
up vote
-1
down vote
favorite
up vote
-1
down vote
favorite
I have a file which is provided below, Col1/Field 1 will always have duplicate data the rest of the fields may/may not but am not worried of other columns, all I need is if there is duplicate data it need to be printed with empty space for col1 and the values of Field 1 will be ordered so no values repeats further down the rows.
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
A1 | 98 | P
A1 | 98 | P
B1 | 98 | P
B1 | 98 | P
B1 | 98 | P
C1 | 98 | P
C1 | 98 | P
C1 | 98 | P
need to convert and the awk/sed need to be applied on col1 only
(the output should look like below after awk/sed/cut is used)
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
awk '!x[$1]++' file <-- is removing whole line
awk/sed/cut anything is fine.
Wrote 50 lines of code , struck at last phase
shell
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
asked Nov 28 at 3:38
vj56789
61
I have a file which is provided below, Col1/Field 1 will always have duplicate data the rest of the fields may/may not but am not worried of other columns, all I need is if there is duplicate data it need to be printed with empty space for col1 and the values of Field 1 will be ordered so no values repeats further down the rows.
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
A1 | 98 | P
A1 | 98 | P
B1 | 98 | P
B1 | 98 | P
B1 | 98 | P
C1 | 98 | P
C1 | 98 | P
C1 | 98 | P
need to convert and the awk/sed need to be applied on col1 only
(the output should look like below after awk/sed/cut is used)
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
awk '!x[$1]++' file <-- is removing whole line
awk/sed/cut anything is fine.
Wrote 50 lines of code , struck at last phase
shell
shell
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
asked Nov 28 at 3:38
vj56789
61
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
asked Nov 28 at 3:38
vj56789
61
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
edited Nov 28 at 6:56
Rui F Ribeiro
38.3k1476127
38.3k1476127
asked Nov 28 at 3:38
vj56789
61
asked Nov 28 at 3:38
vj56789
61
asked Nov 28 at 3:38
vj56789
61
61
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
2
down vote
$ awk -F '|' 'BEGIN { OFS=FS } { c1 = $1 } c1 == prev { gsub(".", " ", $1) } { prev = c1; print }' file
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
The awk program reads |-delimited input data and writes |-delimited output data.
For each line in the input, it extracts the first column into c1 and compares it to the previous first column, prev. If they are the same, the value of the first column is overwritten by spaces (this is what the gsub() does, and it prevents the column width from changing). The recorded "previous first column" is then updated with the value of c1 and the (possibly modified) line is printed.
answered Nov 28 at 7:29
Kusalananda
118k16223361
First off - Thanks your solution worked.
– vj56789
Nov 28 at 15:06
What to do if I wish to do the same comparison with Col1 and Col2
– vj56789
Nov 28 at 15:07
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
$ awk -F '|' 'BEGIN { OFS=FS } { c1 = $1 } c1 == prev { gsub(".", " ", $1) } { prev = c1; print }' file
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
The awk program reads |-delimited input data and writes |-delimited output data.
For each line in the input, it extracts the first column into c1 and compares it to the previous first column, prev. If they are the same, the value of the first column is overwritten by spaces (this is what the gsub() does, and it prevents the column width from changing). The recorded "previous first column" is then updated with the value of c1 and the (possibly modified) line is printed.
answered Nov 28 at 7:29
Kusalananda
118k16223361
First off - Thanks your solution worked.
– vj56789
Nov 28 at 15:06
What to do if I wish to do the same comparison with Col1 and Col2
– vj56789
Nov 28 at 15:07
add a comment |
up vote
2
down vote
$ awk -F '|' 'BEGIN { OFS=FS } { c1 = $1 } c1 == prev { gsub(".", " ", $1) } { prev = c1; print }' file
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
The awk program reads |-delimited input data and writes |-delimited output data.
For each line in the input, it extracts the first column into c1 and compares it to the previous first column, prev. If they are the same, the value of the first column is overwritten by spaces (this is what the gsub() does, and it prevents the column width from changing). The recorded "previous first column" is then updated with the value of c1 and the (possibly modified) line is printed.
answered Nov 28 at 7:29
Kusalananda
118k16223361
First off - Thanks your solution worked.
– vj56789
Nov 28 at 15:06
What to do if I wish to do the same comparison with Col1 and Col2
– vj56789
Nov 28 at 15:07
add a comment |
up vote
2
down vote
up vote
2
down vote
$ awk -F '|' 'BEGIN { OFS=FS } { c1 = $1 } c1 == prev { gsub(".", " ", $1) } { prev = c1; print }' file
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
The awk program reads |-delimited input data and writes |-delimited output data.
For each line in the input, it extracts the first column into c1 and compares it to the previous first column, prev. If they are the same, the value of the first column is overwritten by spaces (this is what the gsub() does, and it prevents the column width from changing). The recorded "previous first column" is then updated with the value of c1 and the (possibly modified) line is printed.
answered Nov 28 at 7:29
Kusalananda
118k16223361
$ awk -F '|' 'BEGIN { OFS=FS } { c1 = $1 } c1 == prev { gsub(".", " ", $1) } { prev = c1; print }' file
COL1 | COL2 | COL3
----------------------
A1 | 98 | P
| 98 | P
| 98 | P
B1 | 98 | P
| 98 | P
| 98 | P
C1 | 98 | P
| 98 | P
| 98 | P
The awk program reads |-delimited input data and writes |-delimited output data.
For each line in the input, it extracts the first column into c1 and compares it to the previous first column, prev. If they are the same, the value of the first column is overwritten by spaces (this is what the gsub() does, and it prevents the column width from changing). The recorded "previous first column" is then updated with the value of c1 and the (possibly modified) line is printed.
answered Nov 28 at 7:29
Kusalananda
118k16223361
answered Nov 28 at 7:29
Kusalananda
118k16223361
answered Nov 28 at 7:29
Kusalananda
118k16223361
answered Nov 28 at 7:29
Kusalananda
118k16223361
118k16223361
First off - Thanks your solution worked.
– vj56789
Nov 28 at 15:06
What to do if I wish to do the same comparison with Col1 and Col2
– vj56789
Nov 28 at 15:07
add a comment |
First off - Thanks your solution worked.
– vj56789
Nov 28 at 15:06
What to do if I wish to do the same comparison with Col1 and Col2
– vj56789
Nov 28 at 15:07
First off - Thanks your solution worked.
– vj56789
Nov 28 at 15:06
First off - Thanks your solution worked.
– vj56789
Nov 28 at 15:06
What to do if I wish to do the same comparison with Col1 and Col2
– vj56789
Nov 28 at 15:07
What to do if I wish to do the same comparison with Col1 and Col2
– vj56789
Nov 28 at 15:07
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f484570%2fconverting-data-and-using-sed-awk%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
