detecting unique lines from log file











up vote
0
down vote

favorite












I have a large log file and would like to detect the patterns instead of specific lines.



for example: 



/path/messages-20181116:11/15/2018 14:23:05.159|worker001|clusterm|I|userx deleted job 5018

/path/messages-20181116:11/15/2018 14:41:25.662|worker001|clusterm|I|userx deleted job 4895

/path/messages-20181116:11/15/2018 14:41:25.673|worker000|clusterm|I|userx deleted job 4890

/path/messages-20181116:11/15/2018 14:41:25.681|worker000|clusterm|I|userx deleted job 4889

11/09/2018 06:18:55.115|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1 

11/09/2018 06:18:55.118|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.120|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.140|scheduler000|clusterm|P|PROF: job dispatching took 5.005 s (10 fast)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 1 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 5 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 3 job(s)             

11/09/2018 06:18:55.145|scheduler000|clusterm|P|PROF: parallel matching   14  0438 107668                 

11/09/2018 06:18:55.148|scheduler000|clusterm|P|PROF: sequential matching  9  0261   8203               

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting :wc =0.006s              

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc=5.005              

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting : wc=0.006s

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc =0.015


becomes something like below: 



/path/messages-*NUMBER*:*DATE* *TIME*|worker001|clusterm|I|userx deleted job *NUMBER*

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job profiling(low job) of *NUMBER* 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job dispatching took *NUMBER* s (*NUMBER* fast)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: dispatched *NUMBER* job(s)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: parallel matching   *NUMBER*  *NUMBER* *NUMBER*                 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: sequential matching  *NUMBER*  *NUMBER*   *NUMBER*               

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job sorting :wc =*NUMBER*s              

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job dispatching: wc=*NUMBER*


which greatly reduce the number of lines and make analyzing/reading log by eye easier.



basically detecting variable words and replace them with some symbol.






command-line logs wildcards text







share|improve this question







New contributor




user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
















  • 1




    What steps have you tried to take on your own? What were the results? Please include this in your question.
    – Panki
    2 days ago










  • have you looked at cut and uniq?
    – ctrl-alt-delor
    2 days ago










  • Uniq will only work with exactly matching lines. Will not work for two same line with different time stamp. Cut you need to read the whole log file and yet you don’t know the patterns
    – user772266
    2 days ago












  • I do used sort -u | uniq but this shows equal lines as if two lines only differ in time stamp both will be printed.
    – user772266
    2 days ago










  • I don't understand the transformations you're expecting. Do you want the dates and times replaced by *DATE* *TIME* or are those placeholders for real values of some sort? What makes a line non-unique?
    – Jeff Schaller
    2 days ago















up vote
0
down vote

favorite












I have a large log file and would like to detect the patterns instead of specific lines.



for example: 



/path/messages-20181116:11/15/2018 14:23:05.159|worker001|clusterm|I|userx deleted job 5018

/path/messages-20181116:11/15/2018 14:41:25.662|worker001|clusterm|I|userx deleted job 4895

/path/messages-20181116:11/15/2018 14:41:25.673|worker000|clusterm|I|userx deleted job 4890

/path/messages-20181116:11/15/2018 14:41:25.681|worker000|clusterm|I|userx deleted job 4889

11/09/2018 06:18:55.115|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1 

11/09/2018 06:18:55.118|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.120|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.140|scheduler000|clusterm|P|PROF: job dispatching took 5.005 s (10 fast)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 1 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 5 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 3 job(s)             

11/09/2018 06:18:55.145|scheduler000|clusterm|P|PROF: parallel matching   14  0438 107668                 

11/09/2018 06:18:55.148|scheduler000|clusterm|P|PROF: sequential matching  9  0261   8203               

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting :wc =0.006s              

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc=5.005              

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting : wc=0.006s

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc =0.015


becomes something like below: 



/path/messages-*NUMBER*:*DATE* *TIME*|worker001|clusterm|I|userx deleted job *NUMBER*

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job profiling(low job) of *NUMBER* 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job dispatching took *NUMBER* s (*NUMBER* fast)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: dispatched *NUMBER* job(s)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: parallel matching   *NUMBER*  *NUMBER* *NUMBER*                 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: sequential matching  *NUMBER*  *NUMBER*   *NUMBER*               

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job sorting :wc =*NUMBER*s              

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job dispatching: wc=*NUMBER*


which greatly reduce the number of lines and make analyzing/reading log by eye easier.



basically detecting variable words and replace them with some symbol.






command-line logs wildcards text







share|improve this question







New contributor




user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
















  • 1




    What steps have you tried to take on your own? What were the results? Please include this in your question.
    – Panki
    2 days ago










  • have you looked at cut and uniq?
    – ctrl-alt-delor
    2 days ago










  • Uniq will only work with exactly matching lines. Will not work for two same line with different time stamp. Cut you need to read the whole log file and yet you don’t know the patterns
    – user772266
    2 days ago












  • I do used sort -u | uniq but this shows equal lines as if two lines only differ in time stamp both will be printed.
    – user772266
    2 days ago










  • I don't understand the transformations you're expecting. Do you want the dates and times replaced by *DATE* *TIME* or are those placeholders for real values of some sort? What makes a line non-unique?
    – Jeff Schaller
    2 days ago













up vote
0
down vote

favorite









up vote
0
down vote

favorite











I have a large log file and would like to detect the patterns instead of specific lines.



for example: 



/path/messages-20181116:11/15/2018 14:23:05.159|worker001|clusterm|I|userx deleted job 5018

/path/messages-20181116:11/15/2018 14:41:25.662|worker001|clusterm|I|userx deleted job 4895

/path/messages-20181116:11/15/2018 14:41:25.673|worker000|clusterm|I|userx deleted job 4890

/path/messages-20181116:11/15/2018 14:41:25.681|worker000|clusterm|I|userx deleted job 4889

11/09/2018 06:18:55.115|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1 

11/09/2018 06:18:55.118|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.120|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.140|scheduler000|clusterm|P|PROF: job dispatching took 5.005 s (10 fast)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 1 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 5 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 3 job(s)             

11/09/2018 06:18:55.145|scheduler000|clusterm|P|PROF: parallel matching   14  0438 107668                 

11/09/2018 06:18:55.148|scheduler000|clusterm|P|PROF: sequential matching  9  0261   8203               

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting :wc =0.006s              

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc=5.005              

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting : wc=0.006s

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc =0.015


becomes something like below: 



/path/messages-*NUMBER*:*DATE* *TIME*|worker001|clusterm|I|userx deleted job *NUMBER*

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job profiling(low job) of *NUMBER* 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job dispatching took *NUMBER* s (*NUMBER* fast)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: dispatched *NUMBER* job(s)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: parallel matching   *NUMBER*  *NUMBER* *NUMBER*                 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: sequential matching  *NUMBER*  *NUMBER*   *NUMBER*               

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job sorting :wc =*NUMBER*s              

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job dispatching: wc=*NUMBER*


which greatly reduce the number of lines and make analyzing/reading log by eye easier.



basically detecting variable words and replace them with some symbol.






command-line logs wildcards text







share|improve this question







New contributor




user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











I have a large log file and would like to detect the patterns instead of specific lines.



for example: 



/path/messages-20181116:11/15/2018 14:23:05.159|worker001|clusterm|I|userx deleted job 5018

/path/messages-20181116:11/15/2018 14:41:25.662|worker001|clusterm|I|userx deleted job 4895

/path/messages-20181116:11/15/2018 14:41:25.673|worker000|clusterm|I|userx deleted job 4890

/path/messages-20181116:11/15/2018 14:41:25.681|worker000|clusterm|I|userx deleted job 4889

11/09/2018 06:18:55.115|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1 

11/09/2018 06:18:55.118|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.120|scheduler000|clusterm|P|PROF: job profiling(low job) of 9473507.1                

11/09/2018 06:18:55.140|scheduler000|clusterm|P|PROF: job dispatching took 5.005 s (10 fast)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 1 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 5 job(s)             

11/09/2018 06:18:55.143|scheduler000|clusterm|P|PROF: dispatched 3 job(s)             

11/09/2018 06:18:55.145|scheduler000|clusterm|P|PROF: parallel matching   14  0438 107668                 

11/09/2018 06:18:55.148|scheduler000|clusterm|P|PROF: sequential matching  9  0261   8203               

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting :wc =0.006s              

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc=5.005              

11/09/2018 06:18:55.561|scheduler000|clusterm|P|PROF(1776285440): job sorting : wc=0.006s

11/09/2018 06:18:55.564|scheduler000|clusterm|P|PROF(1776285440): job dispatching: wc =0.015


becomes something like below: 



/path/messages-*NUMBER*:*DATE* *TIME*|worker001|clusterm|I|userx deleted job *NUMBER*

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job profiling(low job) of *NUMBER* 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: job dispatching took *NUMBER* s (*NUMBER* fast)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: dispatched *NUMBER* job(s)             

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: parallel matching   *NUMBER*  *NUMBER* *NUMBER*                 

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF: sequential matching  *NUMBER*  *NUMBER*   *NUMBER*               

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job sorting :wc =*NUMBER*s              

*DATE* *TIME*|scheduler*NUMBER*|clusterm|P|PROF(*NUMBER*): job dispatching: wc=*NUMBER*


which greatly reduce the number of lines and make analyzing/reading log by eye easier.



basically detecting variable words and replace them with some symbol.






command-line logs wildcards text




command-line logs wildcards text






share|improve this question







New contributor




user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question







New contributor




user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question






New contributor




user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked 2 days ago









user772266

1




1




New contributor




user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






user772266 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.








  • 1




    What steps have you tried to take on your own? What were the results? Please include this in your question.
    – Panki
    2 days ago










  • have you looked at cut and uniq?
    – ctrl-alt-delor
    2 days ago










  • Uniq will only work with exactly matching lines. Will not work for two same line with different time stamp. Cut you need to read the whole log file and yet you don’t know the patterns
    – user772266
    2 days ago












  • I do used sort -u | uniq but this shows equal lines as if two lines only differ in time stamp both will be printed.
    – user772266
    2 days ago










  • I don't understand the transformations you're expecting. Do you want the dates and times replaced by *DATE* *TIME* or are those placeholders for real values of some sort? What makes a line non-unique?
    – Jeff Schaller
    2 days ago














  • 1




    What steps have you tried to take on your own? What were the results? Please include this in your question.
    – Panki
    2 days ago










  • have you looked at cut and uniq?
    – ctrl-alt-delor
    2 days ago










  • Uniq will only work with exactly matching lines. Will not work for two same line with different time stamp. Cut you need to read the whole log file and yet you don’t know the patterns
    – user772266
    2 days ago












  • I do used sort -u | uniq but this shows equal lines as if two lines only differ in time stamp both will be printed.
    – user772266
    2 days ago










  • I don't understand the transformations you're expecting. Do you want the dates and times replaced by *DATE* *TIME* or are those placeholders for real values of some sort? What makes a line non-unique?
    – Jeff Schaller
    2 days ago








1




1




What steps have you tried to take on your own? What were the results? Please include this in your question.
– Panki
2 days ago




What steps have you tried to take on your own? What were the results? Please include this in your question.
– Panki
2 days ago












have you looked at cut and uniq?
– ctrl-alt-delor
2 days ago




have you looked at cut and uniq?
– ctrl-alt-delor
2 days ago












Uniq will only work with exactly matching lines. Will not work for two same line with different time stamp. Cut you need to read the whole log file and yet you don’t know the patterns
– user772266
2 days ago






Uniq will only work with exactly matching lines. Will not work for two same line with different time stamp. Cut you need to read the whole log file and yet you don’t know the patterns
– user772266
2 days ago














I do used sort -u | uniq but this shows equal lines as if two lines only differ in time stamp both will be printed.
– user772266
2 days ago




I do used sort -u | uniq but this shows equal lines as if two lines only differ in time stamp both will be printed.
– user772266
2 days ago












I don't understand the transformations you're expecting. Do you want the dates and times replaced by *DATE* *TIME* or are those placeholders for real values of some sort? What makes a line non-unique?
– Jeff Schaller
2 days ago




I don't understand the transformations you're expecting. Do you want the dates and times replaced by *DATE* *TIME* or are those placeholders for real values of some sort? What makes a line non-unique?
– Jeff Schaller
2 days ago















active

oldest

votes











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});






user772266 is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f482960%2fdetecting-unique-lines-from-log-file%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown






























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes








user772266 is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















user772266 is a new contributor. Be nice, and check out our Code of Conduct.













user772266 is a new contributor. Be nice, and check out our Code of Conduct.












user772266 is a new contributor. Be nice, and check out our Code of Conduct.















 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f482960%2fdetecting-unique-lines-from-log-file%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Entries order in /etc/network/interfaces

Image
4 I have a device with two interfaces, eth0 and eth1. One should get its IP from a DHCP server and the other should get its IP statically. If my /etc/network/interfaces file is configured as follows: auto lo iface lo inet loopback auto eth1 iface eth1 inet dhcp iface eth1 inet6 auto auto eth0 iface eth0 inet static address 10.10.10.10 netmask 255.255.255.0 Everything works great, eth1 gets its IP from the DHCP server, eth0 sets its IP statically from the information given, but if I try to change the order in the file, such as auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 10.10.10.10 netmask 255.255.255.0 auto eth1 iface eth1 inet dhcp iface eth1 inet6 auto In this case, eth1 fails to get an IP. Does the /etc/network/interface care about th...
Read more

新発田市

Image
しばたし 新発田市 市のシンボル・新発田城（二の丸隅櫓） 新発田市旗 1934年7月9日制定 新発田市章 1934年7月9日制定 国 日本 地方 中部地方、北陸地方 甲信越地方、信越地方 都道府県 新潟県 団体コード 15206-4 法人番号 5000020152064 面積 533.10 km 2 総人口 96,267 人 [編集] （推計人口、2018年10月1日） 人口密度 181 人/km 2 隣接自治体 新潟市、阿賀野市、胎内市、北蒲原郡聖籠町、東蒲原郡阿賀町 福島県喜多方市 山形県西置賜郡小国町 市の木 サクラ 市の花 アヤメ 市のシンボル 新発田城 新発田市役所 市長 [編集] 二階堂馨 所在地 〒 957-8686 新潟県新発田市中央町3丁目3番3号 北緯37度56分52.5秒東経139度19分38.2秒 外部リンク 公式ウェブサイト ■ ― 政令指定都市 / ■ ― 市 / ■ ― 町 / ■ ― 村 地理院地図 Google Bing GeoHack MapFan Mapion Yahoo! NAVITIME ゼンリン   表示   ウィキプロジェクト 新発田市中心部周辺の空中写真。 1975年撮影の8枚を合成作成。国土交通省 国土画像情報（カラー空中写真）を基に作成。 新発田市 （しばたし）は、新潟県下越地方にある市である。新潟市への通勤率は16.6%（平成22年国勢調査）。 目次 1 概要 2 地理 2.1 隣接している自治体・行政区 3 地域・市街地構成 4 歴史 4.1 市名の由来 4.2 沿革 5 人口 6 行政 6.1 市長 7 経済 7.1 産業 7.1.1 市内の主な企業 7.1.2 漁業 8 姉妹都市・提携都市 8.1 国内 8.2 海外 9 施設・...
Read more

Grub takes very long (several minutes) to open Menu (in Multi-Boot-System)

Image
0 I have a Multi-Boot-System with 4 different installations of Linux. (1 Manjaro, 3 Mint) spread over 2 hard drives (2 on each). After doing grub-install /dev/sdX and update-grub several times within each installation ... the Grub-menu needs more and more time to open. It started with a few more seconds, now it takes several minutes. I tried to purge and reinstall Grub but this made the problem even worse: apt-get purge grub grub-pc grub-common apt-get install grub-common grub-pc os-prober update-grub What could be causing this problem? linux grub2 multiboot share | improve this question edited 4 mins ago ...
Read more