Commonly, we process JSON data by writing a program to load, deserialize and manipulate this data.

Depending on the programming language, this program may require an additional compilation step before being executed within a terminal. For simple operations, such as filtering and mapping, we don't need to write an additional program to perform these operations on our JSON data. Rather, we can directly manipulate our JSON data within a terminal via the 

 command-line utility, which allows the editing of streamed JSON data without an interactive text editor interface ("

 for JSON"). If you're looking for a tool to retrieve JSON data from an API endpoint, process this data and save the result to a CSV, TSV or JSON file, then 

 easily accomplishes this task in a single-line command. 

Below, I'm going to show you how to process JSON data with 

 website, downloading a prebuilt binary (compatible with your operating system) and executing this binary once the download is complete.

To verify the installation was successful, restart the terminal, and inside of this terminal, enter the command 

For extensive documentation, enter the command 

To get started, let's pretty-print a JSON dataset (with formatting and syntax-highlighting).

 command must be passed a filter as its first argument. A 

 what output should be returned given the input JSON data.

The most basic filter is the pre-defined 

 to do nothing to the input JSON data and return it as is.

 on a JSON dataset, pipe the stringified JSON to 

 command or the JSON response from an API endpoint via the 

 command). If we pipe the JSON response of a cURL command to 

 pretty-prints this response in the terminal.

$ curl 'https://data.cityofnewyork.us/resource/ia2d-e54m.json' | jq .

Suppose we only wanted a single element from the JSON data. To access a single element from a JSON array, pass the array index filter to 

 representing an index value (positive and negative integer).

To access the penultimate (second to last) element:

If the index value is outside of the JSON array's bounds, then no element is returned by the array index filter:

If an index value is omitted, then all of the elements are returned by the array index filter:

 filter can be used on JSON objects to return all top-level values within the object.

$ echo '{"a": 1, "b": 2, "c": { "d": 3 }}' | jq .[]

In case you are unsure whether the input data is not valid JSON, then append a 

 to the empty square brackets to suppress errors.

For example, if the input data is a stringified integer value...

jq: error (at <stdin>:1): Cannot iterate over number (1)

, this error is suppressed as if no error occurred.

Suppose we only wanted a subset of the JSON data. To extract a sub-array from a JSON array, pass the array/string slice filter to 

 representing starting (inclusive) and ending (exclusive) index values respectively (positive and negative integers). It behaves similar to JavaScript's 

To extract all elements but the first element (omit the first element):

To extract all elements but the last element (omit the last element):

 the output of an identity filter to the built-in 

This returns the total number of elements within the JSON array. For our example dataset, the total number of records returned by the NYC Open Data API is 

 function returns the total number of top-level keys within this object.

$ echo '{"a": {"b": 3}, "c": 3}' | jq '. | length'

To retrieve the length of each item of a JSON array, pipe the output of a 

This returns a list of each element's length. For our example dataset, each record contains four pieces of information: the year, the population of NYC for that year, the total number of gallons (in millions) of water consumed by NYC residents per day and the average number of gallons of water consumed by a NYC resident per day.    

 returns the string's length. If an element is a 

$ echo '{"a": "Hello World!", "b": null}' | jq '.[] | length'

To retrieve the top-level keys from JSON, use the built-in 

 function. These keys are returned as an array of strings. Unlike the 

By default, these keys are sorted alphabetically.

$ echo '{"b": "Hello World!", "a": null}' | jq 'keys'

 function does not sort keys alphabetically and returns the keys in their original order. 

$ echo '{"b": "Hello World!", "a": null}' | jq 'keys_unsorted'

For JSON arrays, this function returns a list of indices.

<h1 data-node-type="titlePlaceholder" class="titlePlaceholder">Processing JSON with <code>jq</code></h1><p>Commonly, we process JSON data by writing a program to load, deserialize and manipulate this data.</p><textarea data-iscodemirror="true" data-mode="javascript" data-uniqueid="cm-tgMfXLlqYzX9ydeIdnz12" id="cm-tgMfXLlqYzX9ydeIdnz12">const fetchData = async () =&gt; {
  try {
    const response = await fetch(API_ENDPOINT);
    const data = await response.json();

    return data;
  } catch (err) {
    throw err;
  }
};

fetchData().then(data =&gt; {
  // Manipulate deserialized JSON data.
}).catch(err =&gt; {
  // Handle error.
});</textarea><p>Depending on the programming language, this program may require an additional compilation step before being executed within a terminal. For simple operations, such as filtering and mapping, we don't need to write an additional program to perform these operations on our JSON data. Rather, we can directly manipulate our JSON data within a terminal via the <strong><code><a href="https://stedolan.github.io/jq/" rel="noopener noreferrer nofollow">jq</a></code></strong> command-line utility, which allows the editing of streamed JSON data without an interactive text editor interface ("<code>sed</code> for JSON"). If you're looking for a tool to retrieve JSON data from an API endpoint, process this data and save the result to a CSV, TSV or JSON file, then <code>jq</code> easily accomplishes this task in a single-line command. </p><p>Below, I'm going to show you how to process JSON data with <code>jq</code>.</p><h2 class="heading-anchor-text" id="installation-and-setup"><span>Installation and Setup</span><a class="d-inline" href="#installation-and-setup">#</a></h2><p>Install the <code>jq</code> command-line utility by visiting the <a href="https://stedolan.github.io/jq/" rel="noopener noreferrer nofollow">homepage</a> of the <code>jq</code> website, downloading a prebuilt binary (compatible with your operating system) and executing this binary once the download is complete.</p><p>Alternatively...</p><ul><li><p>For MacOSX users, if you have <a href="https://brew.sh/" rel="noopener noreferrer nofollow">Homebrew</a> installed on your machine, then run the following command to automatically download and install <code>jq</code>:<code>shell
$ brew install jq</code></p></li><li><p>For Windows users, if you have <a href="https://github.com/chocolatey/choco" rel="noopener noreferrer nofollow">Chocolatey NuGet</a> installed on your machine, then run the following command to automatically download and install <code>jq</code>:<code>shell
$ chocolatey install jq</code></p></li><li><p>For Linux users, run the following command to automatically download and install <code>jq</code>:<code>shell
$ sudo apt-get install jq</code></p></li></ul><p>To verify the installation was successful, restart the terminal, and inside of this terminal, enter the command <code>jq</code>.</p><p>This should print an overview of the <code>jq</code> command:</p><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/q7uyl1o4gd6d1cx/Screen%20Shot%202021-03-22%20at%204.30.35%20PM.png" contenteditable="false"><div></div></figure><p>For extensive documentation, enter the command <code>man jq</code>, which summons <strong>man</strong>ual pages for the <code>jq</code> command:</p><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/4023qcpnhu4p4h8/Screen%20Shot%202021-03-22%20at%204.32.34%20PM.png" contenteditable="false"><div></div></figure><h2 class="heading-anchor-text" id="manipulating-json-data"><span>Manipulating JSON Data</span><a class="d-inline" href="#manipulating-json-data">#</a></h2><h3 class="heading-anchor-text" id="pretty-print-json"><span>Pretty-Print JSON</span><a class="d-inline" href="#pretty-print-json">#</a></h3><p>To get started, let's pretty-print a JSON dataset (with formatting and syntax-highlighting).</p><p>The <code>jq</code> command must be passed a filter as its first argument. A <strong>filter</strong> is a program that tells <code>jq</code> what output should be returned given the input JSON data.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-OzpDC3geMKhd4MApJU7eN" id="cm-OzpDC3geMKhd4MApJU7eN">$ jq &lt;filter&gt;</textarea><p>The most basic filter is the pre-defined <strong>identity filter</strong> <code>.</code>, which tells <code>jq</code> to do nothing to the input JSON data and return it as is.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-QHQl9BEzvwl0mezolgh7O" id="cm-QHQl9BEzvwl0mezolgh7O">$ jq .</textarea><p>To run <code>jq</code> on a JSON dataset, pipe the stringified JSON to <code>jq</code> (e.g., the file content of a <code>.json</code> file via the <code>cat</code> command or the JSON response from an API endpoint via the <code>cURL</code> command). If we pipe the JSON response of a cURL command to <code>jq .</code>, then <code>jq</code> pretty-prints this response in the terminal.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-Gjrwo8Lm3R9gUZsutPGvU" id="cm-Gjrwo8Lm3R9gUZsutPGvU">$ curl 'https://data.cityofnewyork.us/resource/ia2d-e54m.json' | jq .</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/3st8og3dzv596io/Screen%20Shot%202021-03-22%20at%203.58.18%20PM.png" contenteditable="false"><div></div></figure><h3 class="heading-anchor-text" id="accessing-elements"><span>Accessing Elements</span><a class="d-inline" href="#accessing-elements">#</a></h3><p>Suppose we only wanted a single element from the JSON data. To access a single element from a JSON array, pass the array index filter to <code>jq</code>, which follows the syntax <code>.[x]</code> with <code>x</code> representing an index value (positive and negative integer).</p><p>To access the first element:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-6OYkNOCo_LTzjvaqv-vVV" id="cm-6OYkNOCo_LTzjvaqv-vVV">$ jq .[0]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/f87pd8zpax0s41c/Screen%20Shot%202021-03-22%20at%205.17.07%20PM.png" contenteditable="false"><div></div></figure><p>To access the last element:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-QdKUvmdI_sYJ9VQaPwYKe" id="cm-QdKUvmdI_sYJ9VQaPwYKe">$ jq .[-1]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/jgvjfdyzz66otul/Screen%20Shot%202021-03-22%20at%205.18.21%20PM.png" contenteditable="false"><div></div></figure><p>To access the penultimate (second to last) element:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-7T3Sl8k_5KcQvFmnKfhAW" id="cm-7T3Sl8k_5KcQvFmnKfhAW">$ jq .[-2]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/jd0ctkidtsxygyi/Screen%20Shot%202021-03-22%20at%205.20.39%20PM.png" contenteditable="false"><div></div></figure><p>To access the element at index <code>3</code>:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-IdRxD7ehygkC4Cp0LWpBU" id="cm-IdRxD7ehygkC4Cp0LWpBU">$ jq .[3]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/ad7t7d95piv6rnu/Screen%20Shot%202021-03-22%20at%205.19.08%20PM.png" contenteditable="false"><div></div></figure><p>If the index value is outside of the JSON array's bounds, then no element is returned by the array index filter:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-CoHYo2wMmA-HpvSTJQ7Ne" id="cm-CoHYo2wMmA-HpvSTJQ7Ne">$ jq .[41]</textarea><p>Here, the dataset only contains <code>41</code> rows. Therefore, any index beyond <code>40</code> causes the filter to return no element.</p><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/vfgeuh12v4r2q01/Screen%20Shot%202021-03-22%20at%205.40.40%20PM.png" contenteditable="false"><div></div></figure><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/4e7x4wb0s2vca5v/Screen%20Shot%202021-03-22%20at%205.41.06%20PM.png" contenteditable="false"><div></div></figure><p>If an index value is omitted, then all of the elements are returned by the array index filter:</p><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/07axfqdaubvm4n1/Screen%20Shot%202021-03-22%20at%205.17.41%20PM.png" contenteditable="false"><div></div></figure><p>Additionally, the <code>.[]</code> filter can be used on JSON objects to return all top-level values within the object.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-htDnfYObmuIn8xQlEc2xN" id="cm-htDnfYObmuIn8xQlEc2xN">$ echo '{"a": 1, "b": 2, "c": { "d": 3 }}' | jq .[]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/hgfwvftliysjl6g/Screen%20Shot%202021-03-22%20at%205.42.54%20PM.png" contenteditable="false"><div></div></figure><p>In case you are unsure whether the input data is not valid JSON, then append a <code>?</code> to the empty square brackets to suppress errors.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-jFHhG1H_vyalX8e3tYtKZ" id="cm-jFHhG1H_vyalX8e3tYtKZ">$ jq .[]?</textarea><p>For example, if the input data is a stringified integer value...</p><p>Without the <code>?</code>, the error <code>jq: error (at &lt;stdin&gt;:1): Cannot iterate over number (1)</code> will be thrown.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-des8ecZkPDI2pXLGrD-aD" id="cm-des8ecZkPDI2pXLGrD-aD">$ echo '1' | jq .[]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/ws4g62cjfn15ugg/Screen%20Shot%202021-03-22%20at%205.46.16%20PM.png" contenteditable="false"><div></div></figure><p>With the <code>?</code>, this error is suppressed as if no error occurred.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-LeNdFULnApg4uzS-ypuGQ" id="cm-LeNdFULnApg4uzS-ypuGQ">$ echo '1' | jq .[]?</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/8nhatgog6ip3mrg/Screen%20Shot%202021-03-22%20at%205.46.41%20PM.png" contenteditable="false"><div></div></figure><h3 class="heading-anchor-text" id="slicing-a-json-array"><span>Slicing a JSON Array</span><a class="d-inline" href="#slicing-a-json-array">#</a></h3><p>Suppose we only wanted a subset of the JSON data. To extract a sub-array from a JSON array, pass the array/string slice filter to <code>jq</code>, which follows the syntax <code>.[x:y]</code> with <code>x</code> and <code>y</code> representing starting (inclusive) and ending (exclusive) index values respectively (positive and negative integers). It behaves similar to JavaScript's <code><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/slice" rel="noopener noreferrer nofollow">.slice()</a></code> method.</p><p>To extract the first element only:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-GkUD7mgOwPJkF9G9inCVO" id="cm-GkUD7mgOwPJkF9G9inCVO">$ jq .[:1]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/9byt4fc8f635cdr/Screen%20Shot%202021-03-22%20at%205.10.50%20PM.png" contenteditable="false"><div></div></figure><p>To extract the last element only:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-NvfSwYtUt8Kqq1fEIJmG2" id="cm-NvfSwYtUt8Kqq1fEIJmG2">$ jq .[-1:]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/56q9hpqkxl7eno5/Screen%20Shot%202021-03-22%20at%205.10.21%20PM.png" contenteditable="false"><div></div></figure><p>To extract all elements but the first element (omit the first element):</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-ehIIGkUmb7rQzi8VIHKJB" id="cm-ehIIGkUmb7rQzi8VIHKJB">$ jq .[1:]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/bvbdcllc9zlmwi9/Screen%20Shot%202021-03-22%20at%205.09.48%20PM.png" contenteditable="false"><div></div></figure><p>To extract all elements but the last element (omit the last element):</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-CnB3p4ZjmJ9Z92AMIVYnG" id="cm-CnB3p4ZjmJ9Z92AMIVYnG">$ jq .[:-1]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/8lvg64r93drog78/Screen%20Shot%202021-03-22%20at%205.09.10%20PM.png" contenteditable="false"><div></div></figure><p>To extract the elements at indices <code>3</code>-<code>5</code>:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-NFtOrPLeTAylzPgb6Z3Me" id="cm-NFtOrPLeTAylzPgb6Z3Me">$ jq .[3:6]</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/4wl8tc2lnmmo8an/Screen%20Shot%202021-03-22%20at%205.08.14%20PM.png" contenteditable="false"><div></div></figure><h3 class="heading-anchor-text" id="length-of-a-json-array"><span>Length of a JSON Array</span><a class="d-inline" href="#length-of-a-json-array">#</a></h3><p>To retrieve the length of a JSON array, <a href="https://stedolan.github.io/jq/manual/#Pipe%3A%7C" rel="noopener noreferrer nofollow">pipe</a> the output of an identity filter to the built-in <strong><code><a href="https://stedolan.github.io/jq/manual/#length" rel="noopener noreferrer nofollow">length</a></code></strong> function:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-ezsQZjDYxh-tiPeKMjxBk" id="cm-ezsQZjDYxh-tiPeKMjxBk">$ jq ". | length"</textarea><p>This returns the total number of elements within the JSON array. For our example dataset, the total number of records returned by the NYC Open Data API is <code>41</code>.</p><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/7msz6ljtv13x4vy/Screen%20Shot%202021-03-25%20at%203.14.12%20PM.png" contenteditable="false"><div></div></figure><p>For a JSON object, the <code>length</code> function returns the total number of top-level keys within this object.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-PQkFDCm_YJIydR8MWVaf_" id="cm-PQkFDCm_YJIydR8MWVaf_">$ echo '{"a": {"b": 3}, "c": 3}' | jq '. | length'</textarea><p>To retrieve the length of each item of a JSON array, pipe the output of a <code>.[]</code> slice filter to the <code>length</code> function:</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-Q5eXMR0MapLV8SYbhs5xL" id="cm-Q5eXMR0MapLV8SYbhs5xL">$ jq ".[] | length"</textarea><p>This returns a list of each element's length. For our example dataset, each record contains four pieces of information: the year, the population of NYC for that year, the total number of gallons (in millions) of water consumed by NYC residents per day and the average number of gallons of water consumed by a NYC resident per day.    </p><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/3qvdpdx5z9e0h3d/Screen%20Shot%202021-03-26%20at%201.44.42%20AM.png" contenteditable="false"><div></div></figure><p>If an element is a string, then <code>length</code> returns the string's length. If an element is a <code>null</code> value, then <code>length</code> returns zero.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-UHa8BXmzYLTxommZd_rbK" id="cm-UHa8BXmzYLTxommZd_rbK">$ echo '{"a": "Hello World!", "b": null}' | jq '.[] | length'</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/j8886nqpy0sscju/Screen%20Shot%202021-03-26%20at%2010.10.40%20PM.png" contenteditable="false"><div></div></figure><h3 class="heading-anchor-text" id="keys-of-a-json-array"><span>Keys of a JSON Array</span><a class="d-inline" href="#keys-of-a-json-array">#</a></h3><p>To retrieve the top-level keys from JSON, use the built-in <strong><code><a href="" rel="noopener noreferrer nofollow">keys</a></code></strong> function. These keys are returned as an array of strings. Unlike the <code>length</code> function, the <code>keys</code> function requires no filter piping. </p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-lJG36tXt1achRsDQVWB1M" id="cm-lJG36tXt1achRsDQVWB1M">$ jq 'keys'</textarea><p>By default, these keys are sorted alphabetically.</p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-xItvtpWrSN2ocGfe7d9RM" id="cm-xItvtpWrSN2ocGfe7d9RM">$ echo '{"b": "Hello World!", "a": null}' | jq 'keys'</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/04n6vy7fuo3t569/Screen%20Shot%202021-03-26%20at%2011.00.17%20PM.png" contenteditable="false"><div></div></figure><p>Alternatively, the <code>keys_unsorted</code> function does not sort keys alphabetically and returns the keys in their original order. </p><textarea data-iscodemirror="true" data-mode="text/x-sh" data-uniqueid="cm-UGYxmVH_Gvp7B3UL1cC7B" id="cm-UGYxmVH_Gvp7B3UL1cC7B">$ echo '{"b": "Hello World!", "a": null}' | jq 'keys_unsorted'</textarea><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/7wninqiokjizcan/Screen%20Shot%202021-03-26%20at%2011.26.14%20PM.png" contenteditable="false"><div></div></figure><p>For JSON arrays, this function returns a list of indices.</p><figure class="photo figure-position-inline"><img src="https://www.dl.dropboxusercontent.com/s/9zogqssn7s5ofoy/Screen%20Shot%202021-03-26%20at%2010.57.34%20PM.png" contenteditable="false"><div></div></figure><h2 class="heading-anchor-text" id="next-steps"><span>Next Steps</span><a class="d-inline" href="#next-steps">#</a></h2><p>Experiment with these techniques on other JSON data sources/files.</p><h2 class="heading-anchor-text" id="sources"><span>Sources</span><a class="d-inline" href="#sources">#</a></h2><ul><li><p><code><a href="https://stedolan.github.io/jq/" rel="noopener noreferrer nofollow">jq</a></code><a href="https://stedolan.github.io/jq/" rel="noopener noreferrer nofollow"> Documentation</a></p></li></ul><p></p>

Learn

The newline Guide to Building Your First GraphQL Server with Node and TypeScript

Teach

Amelia Wattenberger

Author of Fullstack D3

Community

Processing JSON with jq

Tags

Responses (0)

Masterclasses

Tutorials

Fullstack React with TypeScript