filtering pattern in a file using sed or grep











up vote
-2
down vote

favorite












I'd like to store all the unique usernames present in the extended.log file into a new file using the commands awk, grep, and/or sed.



The following are the field-names in my file, separated by tabs. and I just want the values of the "username" field (the 12th field).



"record_id"     "client_id"     "request_id"    "date_time"     "elapsed_time"  "status"        "size"  "upload"        "download"      "bypassed"      "client_ip"     "username"      "method"        "url"   "http_referer"  "useragent"     "mime"  "filter_name"   "filtering_reason"      "interface"     "cachecode"     "peercode"      "peer"  "request_host"  "request_tld"   "referer_host"  "referer_tld"   "range" "time_profiles" "user_groups"   "request_profiles"      "application_signatures"        "categories"    "response_profiles"     "upload_content_types"  "download_content_types"        "profiles"


The following is a sample of the contents of the file:



"SVZerDLJhIj6G3PA.6575.1466420105.346.1837.1"   "1837"  "1"     "20/Jun/2016:16:25:05"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420107.357.1838.1"   "1838"  "1"     "20/Jun/2016:16:25:07"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420109.367.1840.1"   "1840"  "1"     "20/Jun/2016:16:25:09"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420111.377.1841.1"   "1841"  "1"     "20/Jun/2016:16:25:11"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420113.387.1842.1"   "1842"  "1"     "20/Jun/2016:16:25:13"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420115.399.1843.1"   "1843"  "1"     "20/Jun/2016:16:25:15"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420117.410.1844.1"   "1844"  "1"     "20/Jun/2016:16:25:17"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420119.421.1845.1"   "1845"  "1"     "20/Jun/2016:16:25:19"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420121.431.1846.1"   "1846"  "1"     "20/Jun/2016:16:25:21"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420123.445.1847.1"   "1847"  "1"     "20/Jun/2016:16:25:23"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420108.240.1839.1"   "1839"  "1"     "20/Jun/2016:16:25:23"  "15623" "200"   "2826"  "0"     "2826"  "-"     "192.168.0.14"  "anonymous@192.168.0.14""CONNECT"        "connect://livehelp.safesquid.com:443/" "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"    "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "livehelp.safesquid.com"        "livehelp.safesquid.com"        "safesquid.com" "-"     "-"      "1K-10K"        ""      "NO_AUTHENTICATION"     "uncachable request,BUSINESS SITES REQ" ""      "computersandsoftware"  ""      ""      ""      "uncachable"





awk sed grep







share|improve this question




















  • 1




    welcome to U&L. can you show us a sample of log file ? as is, it is impossible to answer the question. Please, edit your post, do not add sample as a comment.
    – Archemar
    Jun 20 '16 at 10:51






  • 1




    awk '{print $12}' | sort | uniq
    – geisshirt
    Jun 20 '16 at 11:06










  • @geisshirt yeah it worked. and how can I do the same using sed and grep commands..?
    – PeeKay
    Jun 20 '16 at 11:09










  • instead of sort and uniq?
    – geisshirt
    Jun 20 '16 at 11:27










  • I mean i need sort and uniq but using grep and sed commands instead of awk :)
    – PeeKay
    Jun 20 '16 at 11:35















up vote
-2
down vote

favorite












I'd like to store all the unique usernames present in the extended.log file into a new file using the commands awk, grep, and/or sed.



The following are the field-names in my file, separated by tabs. and I just want the values of the "username" field (the 12th field).



"record_id"     "client_id"     "request_id"    "date_time"     "elapsed_time"  "status"        "size"  "upload"        "download"      "bypassed"      "client_ip"     "username"      "method"        "url"   "http_referer"  "useragent"     "mime"  "filter_name"   "filtering_reason"      "interface"     "cachecode"     "peercode"      "peer"  "request_host"  "request_tld"   "referer_host"  "referer_tld"   "range" "time_profiles" "user_groups"   "request_profiles"      "application_signatures"        "categories"    "response_profiles"     "upload_content_types"  "download_content_types"        "profiles"


The following is a sample of the contents of the file:



"SVZerDLJhIj6G3PA.6575.1466420105.346.1837.1"   "1837"  "1"     "20/Jun/2016:16:25:05"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420107.357.1838.1"   "1838"  "1"     "20/Jun/2016:16:25:07"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420109.367.1840.1"   "1840"  "1"     "20/Jun/2016:16:25:09"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420111.377.1841.1"   "1841"  "1"     "20/Jun/2016:16:25:11"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420113.387.1842.1"   "1842"  "1"     "20/Jun/2016:16:25:13"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420115.399.1843.1"   "1843"  "1"     "20/Jun/2016:16:25:15"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420117.410.1844.1"   "1844"  "1"     "20/Jun/2016:16:25:17"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420119.421.1845.1"   "1845"  "1"     "20/Jun/2016:16:25:19"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420121.431.1846.1"   "1846"  "1"     "20/Jun/2016:16:25:21"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420123.445.1847.1"   "1847"  "1"     "20/Jun/2016:16:25:23"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420108.240.1839.1"   "1839"  "1"     "20/Jun/2016:16:25:23"  "15623" "200"   "2826"  "0"     "2826"  "-"     "192.168.0.14"  "anonymous@192.168.0.14""CONNECT"        "connect://livehelp.safesquid.com:443/" "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"    "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "livehelp.safesquid.com"        "livehelp.safesquid.com"        "safesquid.com" "-"     "-"      "1K-10K"        ""      "NO_AUTHENTICATION"     "uncachable request,BUSINESS SITES REQ" ""      "computersandsoftware"  ""      ""      ""      "uncachable"





awk sed grep







share|improve this question




















  • 1




    welcome to U&L. can you show us a sample of log file ? as is, it is impossible to answer the question. Please, edit your post, do not add sample as a comment.
    – Archemar
    Jun 20 '16 at 10:51






  • 1




    awk '{print $12}' | sort | uniq
    – geisshirt
    Jun 20 '16 at 11:06










  • @geisshirt yeah it worked. and how can I do the same using sed and grep commands..?
    – PeeKay
    Jun 20 '16 at 11:09










  • instead of sort and uniq?
    – geisshirt
    Jun 20 '16 at 11:27










  • I mean i need sort and uniq but using grep and sed commands instead of awk :)
    – PeeKay
    Jun 20 '16 at 11:35













up vote
-2
down vote

favorite









up vote
-2
down vote

favorite











I'd like to store all the unique usernames present in the extended.log file into a new file using the commands awk, grep, and/or sed.



The following are the field-names in my file, separated by tabs. and I just want the values of the "username" field (the 12th field).



"record_id"     "client_id"     "request_id"    "date_time"     "elapsed_time"  "status"        "size"  "upload"        "download"      "bypassed"      "client_ip"     "username"      "method"        "url"   "http_referer"  "useragent"     "mime"  "filter_name"   "filtering_reason"      "interface"     "cachecode"     "peercode"      "peer"  "request_host"  "request_tld"   "referer_host"  "referer_tld"   "range" "time_profiles" "user_groups"   "request_profiles"      "application_signatures"        "categories"    "response_profiles"     "upload_content_types"  "download_content_types"        "profiles"


The following is a sample of the contents of the file:



"SVZerDLJhIj6G3PA.6575.1466420105.346.1837.1"   "1837"  "1"     "20/Jun/2016:16:25:05"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420107.357.1838.1"   "1838"  "1"     "20/Jun/2016:16:25:07"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420109.367.1840.1"   "1840"  "1"     "20/Jun/2016:16:25:09"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420111.377.1841.1"   "1841"  "1"     "20/Jun/2016:16:25:11"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420113.387.1842.1"   "1842"  "1"     "20/Jun/2016:16:25:13"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420115.399.1843.1"   "1843"  "1"     "20/Jun/2016:16:25:15"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420117.410.1844.1"   "1844"  "1"     "20/Jun/2016:16:25:17"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420119.421.1845.1"   "1845"  "1"     "20/Jun/2016:16:25:19"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420121.431.1846.1"   "1846"  "1"     "20/Jun/2016:16:25:21"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420123.445.1847.1"   "1847"  "1"     "20/Jun/2016:16:25:23"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420108.240.1839.1"   "1839"  "1"     "20/Jun/2016:16:25:23"  "15623" "200"   "2826"  "0"     "2826"  "-"     "192.168.0.14"  "anonymous@192.168.0.14""CONNECT"        "connect://livehelp.safesquid.com:443/" "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"    "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "livehelp.safesquid.com"        "livehelp.safesquid.com"        "safesquid.com" "-"     "-"      "1K-10K"        ""      "NO_AUTHENTICATION"     "uncachable request,BUSINESS SITES REQ" ""      "computersandsoftware"  ""      ""      ""      "uncachable"





awk sed grep







share|improve this question















I'd like to store all the unique usernames present in the extended.log file into a new file using the commands awk, grep, and/or sed.



The following are the field-names in my file, separated by tabs. and I just want the values of the "username" field (the 12th field).



"record_id"     "client_id"     "request_id"    "date_time"     "elapsed_time"  "status"        "size"  "upload"        "download"      "bypassed"      "client_ip"     "username"      "method"        "url"   "http_referer"  "useragent"     "mime"  "filter_name"   "filtering_reason"      "interface"     "cachecode"     "peercode"      "peer"  "request_host"  "request_tld"   "referer_host"  "referer_tld"   "range" "time_profiles" "user_groups"   "request_profiles"      "application_signatures"        "categories"    "response_profiles"     "upload_content_types"  "download_content_types"        "profiles"


The following is a sample of the contents of the file:



"SVZerDLJhIj6G3PA.6575.1466420105.346.1837.1"   "1837"  "1"     "20/Jun/2016:16:25:05"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420107.357.1838.1"   "1838"  "1"     "20/Jun/2016:16:25:07"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420109.367.1840.1"   "1840"  "1"     "20/Jun/2016:16:25:09"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420111.377.1841.1"   "1841"  "1"     "20/Jun/2016:16:25:11"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420113.387.1842.1"   "1842"  "1"     "20/Jun/2016:16:25:13"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420115.399.1843.1"   "1843"  "1"     "20/Jun/2016:16:25:15"  "5"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420117.410.1844.1"   "1844"  "1"     "20/Jun/2016:16:25:17"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420119.421.1845.1"   "1845"  "1"     "20/Jun/2016:16:25:19"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420121.431.1846.1"   "1846"  "1"     "20/Jun/2016:16:25:21"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420123.445.1847.1"   "1847"  "1"     "20/Jun/2016:16:25:23"  "4"     "200"   "0"     "-"     "0"     "-"     "192.168.12.13" "anonymous@192.168.12.13""GET"   "-"     "-"     "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)"   "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "safesquid"      "192.168.14.11:8080"    "-"     "-"     "-"     "0"     ""      "NO_AUTHENTICATION"     ""      ""      ""      ""      ""      ""      ""

"SVZerDLJhIj6G3PA.6575.1466420108.240.1839.1"   "1839"  "1"     "20/Jun/2016:16:25:23"  "15623" "200"   "2826"  "0"     "2826"  "-"     "192.168.0.14"  "anonymous@192.168.0.14""CONNECT"        "connect://livehelp.safesquid.com:443/" "-"     "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"    "-"     "-"     "-"     "192.168.14.11:8080"    "TCP_MISS"      "DIRECT"        "livehelp.safesquid.com"        "livehelp.safesquid.com"        "safesquid.com" "-"     "-"      "1K-10K"        ""      "NO_AUTHENTICATION"     "uncachable request,BUSINESS SITES REQ" ""      "computersandsoftware"  ""      ""      ""      "uncachable"





awk sed grep




awk sed grep






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 24 at 20:16









Rui F Ribeiro

38.3k1475126




38.3k1475126










asked Jun 20 '16 at 10:40









PeeKay

12




12








  • 1




    welcome to U&L. can you show us a sample of log file ? as is, it is impossible to answer the question. Please, edit your post, do not add sample as a comment.
    – Archemar
    Jun 20 '16 at 10:51






  • 1




    awk '{print $12}' | sort | uniq
    – geisshirt
    Jun 20 '16 at 11:06










  • @geisshirt yeah it worked. and how can I do the same using sed and grep commands..?
    – PeeKay
    Jun 20 '16 at 11:09










  • instead of sort and uniq?
    – geisshirt
    Jun 20 '16 at 11:27










  • I mean i need sort and uniq but using grep and sed commands instead of awk :)
    – PeeKay
    Jun 20 '16 at 11:35














  • 1




    welcome to U&L. can you show us a sample of log file ? as is, it is impossible to answer the question. Please, edit your post, do not add sample as a comment.
    – Archemar
    Jun 20 '16 at 10:51






  • 1




    awk '{print $12}' | sort | uniq
    – geisshirt
    Jun 20 '16 at 11:06










  • @geisshirt yeah it worked. and how can I do the same using sed and grep commands..?
    – PeeKay
    Jun 20 '16 at 11:09










  • instead of sort and uniq?
    – geisshirt
    Jun 20 '16 at 11:27










  • I mean i need sort and uniq but using grep and sed commands instead of awk :)
    – PeeKay
    Jun 20 '16 at 11:35








1




1




welcome to U&L. can you show us a sample of log file ? as is, it is impossible to answer the question. Please, edit your post, do not add sample as a comment.
– Archemar
Jun 20 '16 at 10:51




welcome to U&L. can you show us a sample of log file ? as is, it is impossible to answer the question. Please, edit your post, do not add sample as a comment.
– Archemar
Jun 20 '16 at 10:51




1




1




awk '{print $12}' | sort | uniq
– geisshirt
Jun 20 '16 at 11:06




awk '{print $12}' | sort | uniq
– geisshirt
Jun 20 '16 at 11:06












@geisshirt yeah it worked. and how can I do the same using sed and grep commands..?
– PeeKay
Jun 20 '16 at 11:09




@geisshirt yeah it worked. and how can I do the same using sed and grep commands..?
– PeeKay
Jun 20 '16 at 11:09












instead of sort and uniq?
– geisshirt
Jun 20 '16 at 11:27




instead of sort and uniq?
– geisshirt
Jun 20 '16 at 11:27












I mean i need sort and uniq but using grep and sed commands instead of awk :)
– PeeKay
Jun 20 '16 at 11:35




I mean i need sort and uniq but using grep and sed commands instead of awk :)
– PeeKay
Jun 20 '16 at 11:35










1 Answer
1






active

oldest

votes

















up vote
1
down vote













try



 sed -e 's/^.*"([^" ]*)"".*/1/' log | sort | uniq



 egrep -o  '[^"]+@[^"]+' log | sort | uniq


where





  • -o print only matched pattern


  • [^X]+ any number (> 0) of char different from X


please note




  • sed solution relays on a typo/feature in your file (double double quote)

  • grep solution relays on foo@some.where pattern

  • awk (or perl for that matter) is better suited to extract n-th field.






share|improve this answer























    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "106"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














     

    draft saved


    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f290887%2ffiltering-pattern-in-a-file-using-sed-or-grep%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    1
    down vote













    try



     sed -e 's/^.*"([^" ]*)"".*/1/' log | sort | uniq
    

    
 egrep -o  '[^"]+@[^"]+' log | sort | uniq


    where





    • -o print only matched pattern


    • [^X]+ any number (> 0) of char different from X


    please note




    • sed solution relays on a typo/feature in your file (double double quote)

    • grep solution relays on foo@some.where pattern

    • awk (or perl for that matter) is better suited to extract n-th field.






    share|improve this answer



























      up vote
      1
      down vote













      try



       sed -e 's/^.*"([^" ]*)"".*/1/' log | sort | uniq
      

      
 egrep -o  '[^"]+@[^"]+' log | sort | uniq


      where





      • -o print only matched pattern


      • [^X]+ any number (> 0) of char different from X


      please note




      • sed solution relays on a typo/feature in your file (double double quote)

      • grep solution relays on foo@some.where pattern

      • awk (or perl for that matter) is better suited to extract n-th field.






      share|improve this answer

























        up vote
        1
        down vote










        up vote
        1
        down vote









        try



         sed -e 's/^.*"([^" ]*)"".*/1/' log | sort | uniq
        

        
 egrep -o  '[^"]+@[^"]+' log | sort | uniq


        where





        • -o print only matched pattern


        • [^X]+ any number (> 0) of char different from X


        please note




        • sed solution relays on a typo/feature in your file (double double quote)

        • grep solution relays on foo@some.where pattern

        • awk (or perl for that matter) is better suited to extract n-th field.






        share|improve this answer














        try



         sed -e 's/^.*"([^" ]*)"".*/1/' log | sort | uniq
        

        
 egrep -o  '[^"]+@[^"]+' log | sort | uniq


        where





        • -o print only matched pattern


        • [^X]+ any number (> 0) of char different from X


        please note




        • sed solution relays on a typo/feature in your file (double double quote)

        • grep solution relays on foo@some.where pattern

        • awk (or perl for that matter) is better suited to extract n-th field.







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Jun 20 '16 at 14:26

























        answered Jun 20 '16 at 14:20









        Archemar

        19.4k93468




        19.4k93468






























             

            draft saved


            draft discarded



















































             


            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f290887%2ffiltering-pattern-in-a-file-using-sed-or-grep%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            Accessing regular linux commands in Huawei's Dopra Linux

            Image
            4 Backstory: I have a Huawei HG8245 router in my house and I want to change the default username/password. I've tried following this guide but the config file always gets reset. My ISP isn't helping either so I'm looking for other ways to accomplish this. The router has telnet access so this seems like a way to change the credentials. The first shell I gain access to looks like this: WAP> Pressing "?" gives me a list of available commands which I've pasted here, the two interesting ones are "su" and "shell". I can't seem to get su access, even after logging in as root (challenge password prompt, and then nothing happens), but shell gives me a proper linux shell. WAP>shell BusyBox v1.18.4 (2017-12-26 17:06:34 CST) built-in shell (ash) Enter 'hel...
            Read more

            Can't connect RFCOMM socket: Host is down

            Image
            up vote 3 down vote favorite I have bluez 5.4.3 and I am attempting to communicate with a ble device through minicom. These two deceives (my computer and a bluno nano) are alreadt paired and this has been my method so far. #bluetoothctl # pair xx:xx:xx:xx:xx:xx # exit Everything is working well so far then... #sudo rfcomm connect /dev/rfcomm0 xx:xx:xx:xx:xx:xx 1 & [1] 3665 Can't connect RFCOMM socket: Host is down Does anyone know how to resolve this issue? Any help would be very much appreciated. bluetooth bluez share | improve this question asked Feb 9 '17 at 18:46 Brandon Knape 46 2 ...
            Read more

            Kernel panic - not syncing: Fatal Exception in Interrupt

            Image
            0 Every day this error occured on my pc, kernel panic error on interrupt, what will be the problem? linux-mint share asked 6 mins ago Vinod Selvin Vinod Selvin 1 1 New contributor Vinod Selvin is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. add a comment ...
            Read more