parse one field from an JSON array into bash array

Question

I have a JSON output that contains a list of objects stored in a variable. (I may not be phrasing that right)

[
  {
    "item1": "value1",
    "item2": "value2",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "value2_2",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]

I need all the values for item2 in a array for a bash script to be run on ubuntu 14.04.1.

I have found a bunch of ways to get the entire result into an array but not just the items I need

Answer 1

Using jq :

readarray arr < <(jq '.[].item2' json)
printf '%s\n' "${arr[@]}"

If you need a more hardened way:

readarray -td '' arr

for inputs with newlines or other special characters, avoiding word splitting.

Output:

value2
value2_2

Check:

Process Substitution >(command ...) or <(...) is replaced by a temporary filename. Writing or reading that file causes bytes to get piped to the command inside. Often used in combination with file redirection: cmd1 2> >(cmd2). See http://mywiki.wooledge.org/ProcessSubstitution http://mywiki.wooledge.org/BashFAQ/024

Answer 2

The following is actually buggy:

# BAD: Output line of * is replaced with list of local files; can't deal with whitespace
arr=( $( curl -k "$url" | jq -r '.[].item2' ) )

---

If you have bash 4.4 or newer, a best-of-all-worlds option is available:

# BEST: Supports bash 4.4+, with failure detection and newlines in data
{ readarray -t -d '' arr && wait "$!"; } < <(
  set -o pipefail
  curl --fail -k "$url" | jq -j '.[].item2 | (., "\u0000")'
)

...whereas with bash 4.0, you can have terseness at the cost of failure detection and literal newline support:

# OK (with bash 4.0), but can't detect failure and doesn't support values with newlines
readarray -t arr < <(curl -k "$url" | jq -r '.[].item2' )

...or bash 3.x compatibility and failure detection, but without newline support:

# OK: Supports bash 3.x; no support for newlines in values, but can detect failures
IFS=$'\n' read -r -d '' -a arr < <(
  set -o pipefail
  curl --fail -k "$url" | jq -r '.[].item2' && printf '\0'
)

...or bash 3.x compatibility and newline support, but without failure detection:

# OK: Supports bash 3.x and supports newlines in values; does not detect failures
arr=( )
while IFS= read -r -d '' item; do
  arr+=( "$item" )
done < <(curl --fail -k "$url" | jq -j '.[] | (.item2, "\u0000")')

Answer 3

Use jq to produce a shell statement that you evaluate:

eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"

Given the JSON document in your question, the call to jq will produce the string

arr=( 'value2' 'value2_2' )

which is then evaluated by your shell. Evaluating that string will create the named array arr with the two elements value2 and value2_2:

$ eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"
$ printf '"%s"\n' "${arr[@]}"
"value2"
"value2_2"

The @sh operator in jq takes care to properly quote the data for the shell.

Alternatively, move the arr=( ... ) part out of the jq expression:

eval "arr=( $( jq -r '@sh "\([.[].item2])"' file.json ) )"

Now, jq only generates the quoted list of elements, which is then inserted into arr=( ... ) and evaluated.

If you need to read data from a curl command, use curl ... | jq -r ... in place of jq -r ... file.json in the commands above.

Answer 4

To handle arbitrary values:

#!/bin/bash
cat <<EOF >json
[
  {
    "item1": "value1",
    "item2": "  val'\"ue2  ",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "  value\n2_2  ",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]
EOF
eval "arr=( $(jq -r ' .[].item2 | @sh ' json) )"
printf '<%s>\n' "${arr[@]}"

Output:

<  val'"ue2  >
<  value
2_2  >

Putting together some of the other solutions in this question mentioned in comments and answers from @CharlesDuffy, @StéphaneChazelas, @Kusalananda:

#!/bin/bash
v1='a\nb  \n'   
v2='c'\''\"d  '   # v2 will contain <c'\"d  >  
printf '$v1=<%s>\n$v2=<%s>\n\n' "$v1" "$v2"

>json printf "%s\n" "[ \"$v1\", \"$v2\" ]" 
printf 'JSON data: '; cat json
printf '\n'

eval "arr=( $( cat json | jq -r '.[] | @sh ' ) )" 
printf '$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "${arr[@]}"

set -- 
eval "set -- $( cat json | jq -r '[.[]] | @sh ' )" 
printf '$1:<%s>\n$2:<%s>\n\n' "$1" "$2"

{ readarray -td '' arr2 && wait "$!"; } < <( 
   cat json | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "$?" "${arr2[@]}"

{ readarray -td '' arr3 && wait "$!"; } < <( 
   { echo x; cat json; } | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n' "$?"

Output:

$v1=<a\nb  \n>
$v2=<c'\"d  >

JSON data: [ "a\nb  \n", "c'\"d  " ]

$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

$1:<a
b  
>
$2:<c'"d  >

rc=0
$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

parse error: Invalid numeric literal at line 2, column 0
rc=4

Answer 5

Thanks to sputnick I got to this:

arr=( $(curl -k https://localhost/api | jq -r '.[].item2') )

The JSON I have is the output from an API. All I needed to do wans remove the file argument and pipe | the output of curl to jq. Works great and saved some steps.

Answer 6

as an easy alternative, look at jtc tool (at https://github.com/ldn-softdev/jtc), to achieve the same thing (as in jq's example):

bash $ arr=( $(jtc -w '<item2>l+0' file.json) )
bash $ printf '%s\n' "${arr[@]}"
"value2"
"value2_2"
bash $ 

explanation on -w option: angular brackets <...> specify search entire json, suffix l instructs to search labels rather than values, +0 instructs to find all occurrences (rather than just first one).