Conversion
Pretty print numbers
def to_precision($p):
. |tostring|split(".")
| [.[0], (.[1]|split("")|.[0:($p|tonumber)]|join(""))]
| join(".")
| tonumber;
def humansize(bytes;$p):
(bytes|tonumber) as $size |
if $size > 1073741824 then "\(($size/1073741824)|to_precision($p))G"
elif $size > 1048576 then "\(($size/1048576)|to_precision($p))M"
elif $size > 1024 then "\(($size/1024)|to_precision($p))K"
else $size
end;
def humansize(bytes): humansize(bytes;1);
Examples
$ jq -nr 'include "recipes"; humansize(45992)'
44.9K
$ jq -nr 'include "recipes"; humansize(301818073)'
287.8M
$ jq -nr 'include "recipes"; humansize(3018180739)'
2.8G
$ jq -nr 'include "recipes"; humansize(47221;9)'
46.114257812K
Describe
Get object outline
def describe:
walk(
if (type == "object" or type == "array")
then (if (type == "array") then ([limit(1;.[])]) else . end)
else (
if (type == "string") and (test("^https?"))
then "url"
else ((.|fromdate|"date")? // type)
end
)
end
);
$ jq 'include "recipes"; describe' data/sample-github-events.json
[
{
"id": "string",
"type": "string",
"actor": {
"id": "number",
"login": "string",
"display_login": "string",
"gravatar_id": "string",
"url": "url",
"avatar_url": "url"
},
"repo": {
"id": "number",
"name": "string",
"url": "url"
},
"payload": {
"ref": "string",
"ref_type": "string",
"master_branch": "string",
"description": "null",
"pusher_type": "string"
},
"public": "boolean",
"created_at": "date"
}
]
Flatten
Flat JSON
Data: data/sample-github-events.json
def flat_object:
[paths(scalars) as $path
| {"key": $path | join("_"), "value": getpath($path)}]
| from_entries;
def flat_array:
map( flat_object );
jq 'include "recipes";
map(
if type == "object"
then flat_object elif type == "array"
then flat_array
else .
end
)' data/sample-github-events.json
Output
[
{
"id": "30572272710",
"type": "CreateEvent",
"actor_id": 17685332,
"actor_login": "nntrn",
"actor_display_login": "nntrn",
"actor_gravatar_id": "",
"actor_url": "https://api.github.com/users/nntrn",
"actor_avatar_url": "https://avatars.githubusercontent.com/u/17685332?",
"repo_id": 582752600,
"repo_name": "nntrn/jq-recipes",
"repo_url": "https://api.github.com/repos/nntrn/jq-recipes",
"payload_ref": "master",
"payload_ref_type": "branch",
"payload_master_branch": "main",
"payload_pusher_type": "user",
"public": true,
"created_at": "2023-07-21T00:01:11Z"
}
]
Github API
def github_raw_url:
[
"curl --create-dirs -o \(.repository.full_name)/\(.path) ",
(.html_url|gsub("/github.com";"/raw.githubusercontent.com")|gsub("/blob/"; "/")),
(if .repository.private then " -H \"Authorization: Bearer $GITHUB_TOKEN\"" else "" end)
] | join("");
eval "$(
curl -s -H "Authorization: Bearer $GITHUB_TOKEN" 'https://api.github.com/search/code?per_page=3&q=language:shell+user:nntrn+NOT+is:fork' |
jq -r 'include "recipes"; .items|map(github_raw_url)|join("\n")'
)"
Pick
[Source]
# stream should be a stream of dot-paths
def pick(stream):
. as $in
| reduce path(stream) as $a (null;
setpath($a; $in|getpath($a)) );
DATA: simple-nest.json
$ jq 'include "recipes"; pick(.a.d.e)' data/simple-nest.json
{
"a": {
"d": {
"e": 2
}
}
}
DATA: api-extractor.schema.json
$ jq 'include "recipes"; pick(.properties.dtsRollup.type)' data/api-extractor.schema.json
{
"properties": {
"dtsRollup": {
"type": "object"
}
}
}
string pick
def spick($key):
getpath([($key|split(".")[]|select(length > 0))]);
Read history
Compile timestamps in .bash_history for all users
def history:
map(
if test("#[0-9]{10,12}")
then "\(.|gsub("#";"")|tonumber|todate)"
else "\t\(.)\n"
end
) | join("");
Requires root privileges
head -n 100 /home/*/.bash_history |
jq -Rs -r 'include "jqrecipes"; split("\n") | history'
Output:
==> /home/svc_prddbaasawx/.bash_history <==
2022-05-20 17:25:28 cat .git/config
2022-05-27 02:35:56 git --help all
==> /home/admdarrell/.bash_history <==
2022-02-27T07:32:57Z sudo su -
2022-02-27T07:33:21Z exit
2022-02-27T07:35:24Z sudo su -
==> /home/annie_tran/.bash_history <==
2021-03-31 11:50:20 sudo su -
2021-03-31 11:51:25 clear
2021-03-31 11:51:27 pstree
Before:
#1617209420
sudo su -
#1617209485
clear
#1617209487
pstree
Text
Recursively split strings
def split_newlines($s):
if ((type == "string") and (($s|tostring|split("\n")|length) > 1)?)
then ($s|tostring|split("[\\r\\n]+([\\s]+)?";"x"))
elif (type == "object") then to_entries
else $s end;
def recuse_split_newlines: walk(split_newlines(.)|from_entries? // .);
Quoting
def squo: [39]|implode;
def squote($text): [squo,$text,squo]|join("");
def dquote($text): "\"\($text)\"";
def unsmart($text): $text | gsub("[“”]";"\"") | gsub("[’‘]";"'");
def unsmart: . | unsmart;
Cleaning up text
gsub("([^\\w\\d\\s])\\1(\\1)?";"")
$ jq -rR 'gsub("([^\\w\\d\\s])\\1(\\1)?";"")' gsub.txt
Date table sorting|December 31
Charlotte, North Carolina|Charlotte (4)
North Carolina
5
Unroll
[leaf_paths as $path | {
"key": $path | map(tostring) | join("_"),
"value": getpath($path)
}] | from_entries
$ cat data/nested.json|jq '[leaf_paths as $path | {
"key": $path | map(tostring) | join("_"),
"value": getpath($path)
}] | from_entries
'
{
"server_atx_user": "annie",
"server_atx_port": 22,
"storage_nyc_user": "nntrn",
"storage_nyc_port": 22
}
def categorize:
# Returns "object", "array" or "scalar" to indicate the category
# of the piped element.
if type == "object" then "object"
elif type == "array" then "array"
else "scalar"
end;
def pluck($category):
# Plucks the children of a particular category from piped element.
if categorize != "object"
then empty
else to_entries[]
| select(.value | categorize == $category)
| .value
end;
def split:
# Splits the piped element's children into arrays, scalars, and objects
# and returns a meta object containing the children seperated by these
# keys. If the piped element is a scalar or array, this does not look
# at the children, but just returns that element in the meta object.
if categorize == "scalar" then { objects: [], arrays: [], scalars: [.] }
elif categorize == "array" then { objects: [], arrays: [.], scalars: [] }
else { objects: [pluck("object")], arrays : [pluck("array")], scalars: [pluck("scalar")] }
end;
def unwrap:
# Unwraps an array recursively, until the elements of the returned array
# are either scalars or objects but not arrays. If piped element is not
# an array, returns the element as is.
if type != "array" then .
elif length == 0 then empty
else .[] | unwrap
end;
def extract($category):
# Extracts the elements of a particular category from the piped in array.
# If the piped in element is not an array, this fn acts as filter to
# only return the element if it is of the desired category.
unwrap | select(.| categorize == $category);
def unroll:
# Unrolls the passed in object recursively until only scalars are left.
# Returns a row for each leaf node of tree structure of the object and
# elements of the row would be all the scalars encountered at all the
# ancestor levels of this left node.
. | .result += .state.scalars
| .state.objects += [.state.arrays | extract("object")]
| .state.objects += [.state.arrays | extract("scalar")]
| if (.state.objects | length == 0 )
then .result
else ({ data : .state.objects,
state: .state.objects[] | split,
result: .result
} | unroll)
end;
def unrolls($data): { data: $data, state: $data| split, result: [] } | unroll ;
def unrolls: unrolls(.);