Slicing JSON with JQ to Create Arrays for New Relic Ingestion











up vote
0
down vote

favorite












sorry for the long title.



I need to take some JSON, slice it using JQ so New Relic can parse it properly as it only takes a specific format when pushing JSON to their dashboards.



I am using a bash script and WPCLI and then formatting the JSONs output using JQ.



Below is my script.



#!/bin/bash

#for d in /var/www/* ; do
# echo "$d"
#done

for f in /var/www/*/public_html/; do
if [[ -d $f ]]; then
#echo "$f"

cd $f;
SITEURL="$(wp option get siteurl)"
USERS="$(wp user list --fields=display_name,user_email,user_registered,roles --format=json | jq .)"
PLUGINS="$(wp plugin list --format=json | jq .)"

echo "${SITEURL}" "${USERS}" "${PLUGINS}"
cd ../.. ;
fi
done


Below is the output I am getting...



cat wp-info-output 
http://mytest1.com {
"display_name": "testuser1",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:09",
"roles": "administrator"
} {
"name": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
}
{
"name": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}


I need it to look like this...



[
{
"eventType":"WordpressSite",
"siteurl":"http://mytest1.com"
},
{
"eventType":"WordpressPlugins",
"pluginName": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
},
{
"eventType":"WordpressPlugins",
"pluginName": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}
{
"eventType":"Users",
"display_name": "testuser2",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:04",
"roles": "administrator"
}
]


Thanks in advance for any tips. I think jq is my answer for manipulating the JSON to be formatted but I am hitting some walls.










share|improve this question
























  • Note that your directory variable $f holds an absolute path, so there's really no need to cd ../.. in the loop
    – glenn jackman
    Nov 30 at 20:44















up vote
0
down vote

favorite












sorry for the long title.



I need to take some JSON, slice it using JQ so New Relic can parse it properly as it only takes a specific format when pushing JSON to their dashboards.



I am using a bash script and WPCLI and then formatting the JSONs output using JQ.



Below is my script.



#!/bin/bash

#for d in /var/www/* ; do
# echo "$d"
#done

for f in /var/www/*/public_html/; do
if [[ -d $f ]]; then
#echo "$f"

cd $f;
SITEURL="$(wp option get siteurl)"
USERS="$(wp user list --fields=display_name,user_email,user_registered,roles --format=json | jq .)"
PLUGINS="$(wp plugin list --format=json | jq .)"

echo "${SITEURL}" "${USERS}" "${PLUGINS}"
cd ../.. ;
fi
done


Below is the output I am getting...



cat wp-info-output 
http://mytest1.com {
"display_name": "testuser1",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:09",
"roles": "administrator"
} {
"name": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
}
{
"name": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}


I need it to look like this...



[
{
"eventType":"WordpressSite",
"siteurl":"http://mytest1.com"
},
{
"eventType":"WordpressPlugins",
"pluginName": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
},
{
"eventType":"WordpressPlugins",
"pluginName": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}
{
"eventType":"Users",
"display_name": "testuser2",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:04",
"roles": "administrator"
}
]


Thanks in advance for any tips. I think jq is my answer for manipulating the JSON to be formatted but I am hitting some walls.










share|improve this question
























  • Note that your directory variable $f holds an absolute path, so there's really no need to cd ../.. in the loop
    – glenn jackman
    Nov 30 at 20:44













up vote
0
down vote

favorite









up vote
0
down vote

favorite











sorry for the long title.



I need to take some JSON, slice it using JQ so New Relic can parse it properly as it only takes a specific format when pushing JSON to their dashboards.



I am using a bash script and WPCLI and then formatting the JSONs output using JQ.



Below is my script.



#!/bin/bash

#for d in /var/www/* ; do
# echo "$d"
#done

for f in /var/www/*/public_html/; do
if [[ -d $f ]]; then
#echo "$f"

cd $f;
SITEURL="$(wp option get siteurl)"
USERS="$(wp user list --fields=display_name,user_email,user_registered,roles --format=json | jq .)"
PLUGINS="$(wp plugin list --format=json | jq .)"

echo "${SITEURL}" "${USERS}" "${PLUGINS}"
cd ../.. ;
fi
done


Below is the output I am getting...



cat wp-info-output 
http://mytest1.com {
"display_name": "testuser1",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:09",
"roles": "administrator"
} {
"name": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
}
{
"name": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}


I need it to look like this...



[
{
"eventType":"WordpressSite",
"siteurl":"http://mytest1.com"
},
{
"eventType":"WordpressPlugins",
"pluginName": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
},
{
"eventType":"WordpressPlugins",
"pluginName": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}
{
"eventType":"Users",
"display_name": "testuser2",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:04",
"roles": "administrator"
}
]


Thanks in advance for any tips. I think jq is my answer for manipulating the JSON to be formatted but I am hitting some walls.










share|improve this question















sorry for the long title.



I need to take some JSON, slice it using JQ so New Relic can parse it properly as it only takes a specific format when pushing JSON to their dashboards.



I am using a bash script and WPCLI and then formatting the JSONs output using JQ.



Below is my script.



#!/bin/bash

#for d in /var/www/* ; do
# echo "$d"
#done

for f in /var/www/*/public_html/; do
if [[ -d $f ]]; then
#echo "$f"

cd $f;
SITEURL="$(wp option get siteurl)"
USERS="$(wp user list --fields=display_name,user_email,user_registered,roles --format=json | jq .)"
PLUGINS="$(wp plugin list --format=json | jq .)"

echo "${SITEURL}" "${USERS}" "${PLUGINS}"
cd ../.. ;
fi
done


Below is the output I am getting...



cat wp-info-output 
http://mytest1.com {
"display_name": "testuser1",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:09",
"roles": "administrator"
} {
"name": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
}
{
"name": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}


I need it to look like this...



[
{
"eventType":"WordpressSite",
"siteurl":"http://mytest1.com"
},
{
"eventType":"WordpressPlugins",
"pluginName": "akismet",
"status": "inactive",
"update": "available",
"version": "4.0.8"
},
{
"eventType":"WordpressPlugins",
"pluginName": "hello",
"status": "inactive",
"update": "none",
"version": "1.7"
}
{
"eventType":"Users",
"display_name": "testuser2",
"user_email": "test.user@testing.com",
"user_registered": "2018-11-26 17:44:04",
"roles": "administrator"
}
]


Thanks in advance for any tips. I think jq is my answer for manipulating the JSON to be formatted but I am hitting some walls.







linux text-processing json jq wordpress






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 30 at 17:49









Jeff Schaller

37k1052121




37k1052121










asked Nov 30 at 17:48









Jovan Hernandez

133




133












  • Note that your directory variable $f holds an absolute path, so there's really no need to cd ../.. in the loop
    – glenn jackman
    Nov 30 at 20:44


















  • Note that your directory variable $f holds an absolute path, so there's really no need to cd ../.. in the loop
    – glenn jackman
    Nov 30 at 20:44
















Note that your directory variable $f holds an absolute path, so there's really no need to cd ../.. in the loop
– glenn jackman
Nov 30 at 20:44




Note that your directory variable $f holds an absolute path, so there's really no need to cd ../.. in the loop
– glenn jackman
Nov 30 at 20:44










1 Answer
1






active

oldest

votes

















up vote
1
down vote













It seems you want to take all those JSON objects and smush them together in one bit array: try



siteurl=$(wp option get siteurl)
users=$(
wp user list --fields=display_name,user_email,user_registered,roles --format=json) |
jq '. | .eventType = "WordpressUser"'
)
plugins=$( wp plugin list --format=json | jq '. | .eventType = "WordpressPlugin"' )

{
printf '{"eventType":"WordpressSite","siteurl":"%s"}n' "$(wp option get siteurl)"
echo "$plugins"
echo "$users"
} | jq -s .





share|improve this answer























  • Yea I know it seems goofy but that's how New Relic want's their json events formatted. Each JSON event should be it's own object, without any real arrays. Therefore, each event would need to be its own object with a header event type. You can see more here: docs.newrelic.com/docs/insights/insights-data-sources/… But I pasted the example New Relic wants.
    – Jovan Hernandez
    Nov 30 at 21:27












  • Also, what you provided got me 99% there! The only thing is the eventType for the plugins is at the bottom of the JSON object instead of at the top.
    – Jovan Hernandez
    Nov 30 at 21:41










  • The order should not matter. The attributes of a JSON object are not inherently ordered.
    – glenn jackman
    Nov 30 at 21:42












  • Ah you're right, it really shouldn't matter. For some reason the Users are not being outputted but I think you got me 99% there and I can continue to hack it out. Thanks!
    – Jovan Hernandez
    Nov 30 at 21:46











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f485212%2fslicing-json-with-jq-to-create-arrays-for-new-relic-ingestion%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
1
down vote













It seems you want to take all those JSON objects and smush them together in one bit array: try



siteurl=$(wp option get siteurl)
users=$(
wp user list --fields=display_name,user_email,user_registered,roles --format=json) |
jq '. | .eventType = "WordpressUser"'
)
plugins=$( wp plugin list --format=json | jq '. | .eventType = "WordpressPlugin"' )

{
printf '{"eventType":"WordpressSite","siteurl":"%s"}n' "$(wp option get siteurl)"
echo "$plugins"
echo "$users"
} | jq -s .





share|improve this answer























  • Yea I know it seems goofy but that's how New Relic want's their json events formatted. Each JSON event should be it's own object, without any real arrays. Therefore, each event would need to be its own object with a header event type. You can see more here: docs.newrelic.com/docs/insights/insights-data-sources/… But I pasted the example New Relic wants.
    – Jovan Hernandez
    Nov 30 at 21:27












  • Also, what you provided got me 99% there! The only thing is the eventType for the plugins is at the bottom of the JSON object instead of at the top.
    – Jovan Hernandez
    Nov 30 at 21:41










  • The order should not matter. The attributes of a JSON object are not inherently ordered.
    – glenn jackman
    Nov 30 at 21:42












  • Ah you're right, it really shouldn't matter. For some reason the Users are not being outputted but I think you got me 99% there and I can continue to hack it out. Thanks!
    – Jovan Hernandez
    Nov 30 at 21:46















up vote
1
down vote













It seems you want to take all those JSON objects and smush them together in one bit array: try



siteurl=$(wp option get siteurl)
users=$(
wp user list --fields=display_name,user_email,user_registered,roles --format=json) |
jq '. | .eventType = "WordpressUser"'
)
plugins=$( wp plugin list --format=json | jq '. | .eventType = "WordpressPlugin"' )

{
printf '{"eventType":"WordpressSite","siteurl":"%s"}n' "$(wp option get siteurl)"
echo "$plugins"
echo "$users"
} | jq -s .





share|improve this answer























  • Yea I know it seems goofy but that's how New Relic want's their json events formatted. Each JSON event should be it's own object, without any real arrays. Therefore, each event would need to be its own object with a header event type. You can see more here: docs.newrelic.com/docs/insights/insights-data-sources/… But I pasted the example New Relic wants.
    – Jovan Hernandez
    Nov 30 at 21:27












  • Also, what you provided got me 99% there! The only thing is the eventType for the plugins is at the bottom of the JSON object instead of at the top.
    – Jovan Hernandez
    Nov 30 at 21:41










  • The order should not matter. The attributes of a JSON object are not inherently ordered.
    – glenn jackman
    Nov 30 at 21:42












  • Ah you're right, it really shouldn't matter. For some reason the Users are not being outputted but I think you got me 99% there and I can continue to hack it out. Thanks!
    – Jovan Hernandez
    Nov 30 at 21:46













up vote
1
down vote










up vote
1
down vote









It seems you want to take all those JSON objects and smush them together in one bit array: try



siteurl=$(wp option get siteurl)
users=$(
wp user list --fields=display_name,user_email,user_registered,roles --format=json) |
jq '. | .eventType = "WordpressUser"'
)
plugins=$( wp plugin list --format=json | jq '. | .eventType = "WordpressPlugin"' )

{
printf '{"eventType":"WordpressSite","siteurl":"%s"}n' "$(wp option get siteurl)"
echo "$plugins"
echo "$users"
} | jq -s .





share|improve this answer














It seems you want to take all those JSON objects and smush them together in one bit array: try



siteurl=$(wp option get siteurl)
users=$(
wp user list --fields=display_name,user_email,user_registered,roles --format=json) |
jq '. | .eventType = "WordpressUser"'
)
plugins=$( wp plugin list --format=json | jq '. | .eventType = "WordpressPlugin"' )

{
printf '{"eventType":"WordpressSite","siteurl":"%s"}n' "$(wp option get siteurl)"
echo "$plugins"
echo "$users"
} | jq -s .






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 30 at 21:43

























answered Nov 30 at 20:42









glenn jackman

49.6k569106




49.6k569106












  • Yea I know it seems goofy but that's how New Relic want's their json events formatted. Each JSON event should be it's own object, without any real arrays. Therefore, each event would need to be its own object with a header event type. You can see more here: docs.newrelic.com/docs/insights/insights-data-sources/… But I pasted the example New Relic wants.
    – Jovan Hernandez
    Nov 30 at 21:27












  • Also, what you provided got me 99% there! The only thing is the eventType for the plugins is at the bottom of the JSON object instead of at the top.
    – Jovan Hernandez
    Nov 30 at 21:41










  • The order should not matter. The attributes of a JSON object are not inherently ordered.
    – glenn jackman
    Nov 30 at 21:42












  • Ah you're right, it really shouldn't matter. For some reason the Users are not being outputted but I think you got me 99% there and I can continue to hack it out. Thanks!
    – Jovan Hernandez
    Nov 30 at 21:46


















  • Yea I know it seems goofy but that's how New Relic want's their json events formatted. Each JSON event should be it's own object, without any real arrays. Therefore, each event would need to be its own object with a header event type. You can see more here: docs.newrelic.com/docs/insights/insights-data-sources/… But I pasted the example New Relic wants.
    – Jovan Hernandez
    Nov 30 at 21:27












  • Also, what you provided got me 99% there! The only thing is the eventType for the plugins is at the bottom of the JSON object instead of at the top.
    – Jovan Hernandez
    Nov 30 at 21:41










  • The order should not matter. The attributes of a JSON object are not inherently ordered.
    – glenn jackman
    Nov 30 at 21:42












  • Ah you're right, it really shouldn't matter. For some reason the Users are not being outputted but I think you got me 99% there and I can continue to hack it out. Thanks!
    – Jovan Hernandez
    Nov 30 at 21:46
















Yea I know it seems goofy but that's how New Relic want's their json events formatted. Each JSON event should be it's own object, without any real arrays. Therefore, each event would need to be its own object with a header event type. You can see more here: docs.newrelic.com/docs/insights/insights-data-sources/… But I pasted the example New Relic wants.
– Jovan Hernandez
Nov 30 at 21:27






Yea I know it seems goofy but that's how New Relic want's their json events formatted. Each JSON event should be it's own object, without any real arrays. Therefore, each event would need to be its own object with a header event type. You can see more here: docs.newrelic.com/docs/insights/insights-data-sources/… But I pasted the example New Relic wants.
– Jovan Hernandez
Nov 30 at 21:27














Also, what you provided got me 99% there! The only thing is the eventType for the plugins is at the bottom of the JSON object instead of at the top.
– Jovan Hernandez
Nov 30 at 21:41




Also, what you provided got me 99% there! The only thing is the eventType for the plugins is at the bottom of the JSON object instead of at the top.
– Jovan Hernandez
Nov 30 at 21:41












The order should not matter. The attributes of a JSON object are not inherently ordered.
– glenn jackman
Nov 30 at 21:42






The order should not matter. The attributes of a JSON object are not inherently ordered.
– glenn jackman
Nov 30 at 21:42














Ah you're right, it really shouldn't matter. For some reason the Users are not being outputted but I think you got me 99% there and I can continue to hack it out. Thanks!
– Jovan Hernandez
Nov 30 at 21:46




Ah you're right, it really shouldn't matter. For some reason the Users are not being outputted but I think you got me 99% there and I can continue to hack it out. Thanks!
– Jovan Hernandez
Nov 30 at 21:46


















draft saved

draft discarded




















































Thanks for contributing an answer to Unix & Linux Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f485212%2fslicing-json-with-jq-to-create-arrays-for-new-relic-ingestion%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Accessing regular linux commands in Huawei's Dopra Linux

Can't connect RFCOMM socket: Host is down

Kernel panic - not syncing: Fatal Exception in Interrupt