How to use jq command (json data processing, formatting)

2022/11/28

The jq command provides a convenient way to extract desired information from JSON data. In this section, we will review the basic usage of the jq command.

TOC

Install

brew install jq

sudo yum -y install epel-release
sudo yum -y install jq

sudo apt -y update
sudo apt -y install jq

JSON data used for confirmation

Here, the following JSON data will be used to check the operation.

{
  "total_count": 3,
  "items": [
    {
      "id": 111,
      "name": "aaa",
      "owner": {
        "id": 1111111,
        "type": "Organization"
      },
      "size": 10
    },
    {
      "id": 222,
      "name": "bbb",
      "owner": {
        "id": 2222222,
        "type": "User"
      },
      "size": 30
    },
    {
      "id": 333,
      "name": "ccc",
      "owner": {
        "id": 3333333,
        "type": "Organization"
      },
      "size": 25
    }
  ]
}

operation check

Get everything from the starting point & reformat

To retrieve all JSON data, use ..

$ cat tmp.json | jq .
{
  "total_count": 3,
  "items": [
    {
      "id": 111,
      "name": "aaa",
      "owner": {
        "id": 1111111,
        "type": "Organization"
      },
      "size": 10
    },
    {
      "id": 222,
      "name": "bbb",
      "owner": {
        "id": 2222222,
        "type": "User"
      },
      "size": 30
    },
    {
      "id": 333,
      "name": "ccc",
      "owner": {
        "id": 3333333,
        "type": "Organization"
      },
      "size": 25
    }
  ]
}

The data is then formatted in an easy-to-read format, as shown below, and can be used as-is for formatting JSON data obtained with curl.

$ echo '{"xx": 123, "y": 234, "z": 345}' | jq .
{
  "xx": 123,
  "y": 234,
  "z": 345
}

Conversely, use the -c option if you do not want the formatting

$ cat tmp.json | jq -c .
{"total_count":3,"items":[{"id":111,"name":"aaa","owner":{"id":1111111,"type":"Organization"},"size":10},{"id":222,"name":"bbb","owner":{"id":2222222,"type":"User"},"size":30},{"id":333,"name":"ccc","owner":{"id":3333333,"type":"Organization"},"size":25}]}

reference
- https://stedolan.github.io/jq/manual/#Identity:.

Specific element acquisition

Get the value of total_count.

$ cat tmp.json | jq '.total_count'
3

Obtaining a specific element in an array

Obtains the value of id from all elements of the items array.

$ cat tmp.json | jq '.items[].id'
111
222
333

Obtains the id value of the first element in the items array.

$ cat tmp.json | jq '.items[0].id'
111

Obtaining the number of array elements

$ cat tmp.json | jq '[.items[]] | length'
3

Remove double quotes

$ cat tmp.json | jq '.items[].name'
"aaa"
"bbb"
"ccc"

The -r option removes double quotes.

$ cat tmp.json | jq -r '.items[].name'
aaa
bbb
ccc

Cast & String Concatenate

The (.id|tostring) part casts number to string.

$ cat tmp.json | jq -r '.items[] | .result = (.id|tostring) + " " + .name'
{
  "id": 111,
  "name": "aaa",
  "owner": {
    "id": 1111111,
    "type": "Organization"
  },
  "size": 10,
  "result": "111 aaa"
}
{
  "id": 222,
  "name": "bbb",
  "owner": {
    "id": 2222222,
    "type": "User"
  },
  "size": 30,
  "result": "222 bbb"
}
{
  "id": 333,
  "name": "ccc",
  "owner": {
    "id": 3333333,
    "type": "Organization"
  },
  "size": 25,
  "result": "333 ccc"
}

If you only want the concatenated string, do the following

$ cat tmp.json | jq -r '.items[] | .result = (.id|tostring) + " " + .name | .result'
111 aaa
222 bbb
333 ccc

map creates a new array

$ cat tmp.json | jq -r '.items | map({ name: .name, owner_id: .owner.id })'
[
  {
    "name": "aaa",
    "owner_id": 1111111
  },
  {
    "name": "bbb",
    "owner_id": 2222222
  },
  {
    "name": "ccc",
    "owner_id": 3333333
  }
]

Similar results can be obtained by doing the following

$ cat tmp.json | jq -r '[.items[] | { name: .name, owner_id: .owner.id }]'
[
  {
    "name": "aaa",
    "owner_id": 1111111
  },
  {
    "name": "bbb",
    "owner_id": 2222222
  },
  {
    "name": "ccc",
    "owner_id": 3333333
  }
]

aggregation

Calculates the sum of each size in the items array.

$ cat tmp.json | jq -r '[.items[].size] | add'
65

Similar results can be obtained by doing the following

$ cat tmp.json | jq 'reduce .items[] as $item (0; . + $item.size)'
65

narrowing-down

Extracts elements from the items array whose size is greater than 25.

$ cat tmp.json | jq '[.items[]]' | jq 'map(select( .size > 25 ))'
[
  {
    "id": 222,
    "name": "bbb",
    "owner": {
      "id": 2222222,
      "type": "User"
    },
    "size": 30
  }
]

Extracts elements from the items array whose owner.type value is Organization.

$ cat tmp.json | jq '[.items[]]' | jq 'map(select( .owner.type == "Organization" ))'
[
  {
    "id": 111,
    "name": "aaa",
    "owner": {
      "id": 1111111,
      "type": "Organization"
    },
    "size": 10
  },
  {
    "id": 333,
    "name": "ccc",
    "owner": {
      "id": 3333333,
      "type": "Organization"
    },
    "size": 25
  }
]

CSV output

Use @csv to output in CSV format.

$ cat tmp.json | jq -r '.items[] | [.name, .owner.id] | @csv'
"aaa",1111111
"bbb",2222222
"ccc",3333333

reference

Let's share this post !