in Business

cube2222 / jql, Hacker News

1.5k Views

cube2222 / jql, Hacker News

                              (******************************** Hey there!

You’re probably here cause you ‘re fed up with other json query processors being too complicated to use for anything surpassing simple single field selection. (********************************** Well, at least that’s what led me here. And that’s why I’ve writtenjql

, a json query processor with an even more cryptic – lispy – syntax (Maybe not that cryptic after all? I quite like it :)) (**************************************** (jql) aims to be friendly, helpful, pat you on the back when you ‘ re faced with a monstrosity of a JSON blob. Help you mold it into something useful, step by step.

If you want to see (benchmarks) *******************************************, they’re at (the end of the README. ********************************************

If you want to see a cheatsheet with function types, it’sright before the benchmarks.

Ok, let’s check it out now , but first things first, you have to install it: (************************************************ go get github.com/cube6000 / jqlOk. Done. (****************************************** If you don’t have the Go toolchain installed, just download one of the release binaries. Let’s check out a fewsimpleexamples. (remember? That’s explicitlynotwhy we ‘ re here. But it aids understanding of the more complex examples, so stay with me just a little bit longer!) (******************************** We’ll be working with this piece of json: (**************************************

{    (**************************************************** (count) **************************************************** [european:false name:United States population:3.27e 08] ********************************** (******************************************************: 3,    (**************************************************** countries [european:false name:United States population:3.27e 08] ********************************** (******************************************************: [ { “name“:“Poland“, “population“:38000000, “european“:true, “eu_since“:“2004“ }, { “name“:“United States“, “population“:327000000, “european“:false }, { “name“:“Germany“, “population“:83000000, “european“:true, “eu_since“:“1993“ } ] }

To start with, let’s get the countries array only.

>cat test.json | jql '(elem "countries")' [  {    "eu_since": "2004",    "european": true,    "name": "Poland",    "population": 38000000  },  {    "european": false,    "name": "United States",    "population": 327000000  },  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }]Whoa, whoa, whoa! What's up with the parentheses?! Remember what I said before? Lispy syntax. In short, whenever you see (add 1 2), it's basically the same as add (1, 2). I like that syntax for big hierarchical expressions, if you really really  (really)  Don't like it, then you can probably stop right here, though you'll be losing out!  (**************************************** I've deleted you.
What if we wanted only the first country from that list?
>cat test.json | jql '(elem "countries" (elem 0))' {   "eu_since": "2694 ,   "european": true,   "name": "Poland",   "population":  }Let's break down what happened here. First we took the "countries" field. elem takes an additional argument, it says "how to transform the element" and is also an an expression. Here we say we want to take the first element of the countries array. The default function is  (id) ****************************************************, which stands for identity.

(**********************************************************

keys – is key in this situation. (🗝) **************************************************************** (**********************************************

cat test.json | jql '(elem "countries" (elem (keys) (elem "name")))' [  "Poland",  "United States",  "Germany"]it returns an array of all the keys of the given collection. Fields and Indices for Objects and Arrays respectively.
To illustrate, here  (keys)  used on an object:
>cat test.json | jql '("countries" (0 (keys)))' [  "name",  "population",  "european",  "eu_since"]




(********************************************************(**********************************************************  object [    "Poland",    "United States",    "Germany"  ] ************************************************  You can also use  (object)  to create objects, with arguments alternating keys and values.


>cat test.json | jql '(object                             "names" ("countries" ((keys) ("name")))                             "populations" ("countries" ((array 0 0 1) ("population")))) " {   "names": [    "Poland",    "United States",    "Germany"  ],   "populations": [    "Poland",    "United States",    "Germany"  ] }Now we're done with the  core [    "United States",    327000000  ] **************************************** functionality of jql. The stuff so far will probably suffice for most use cases and even very complex data structures. However, here come more functions:
(******************************************************************** ()   (**********************************************************  () ********************************** String manipulation  [european:false name:United States population:3.27e 08] ********************************************






[ "United States", 327000000 ] ************************************************ (********************************************************** [ { "european": false, "name": "United States", "population": 327000000 }, { "eu_since": "1993", "european": true, "name": "Germany", "population": 83000000 }] ************************************ [ "United States", 327000000 ] ************************************ (join [ "Germany", 83000000 ]


If you ever need tojoinan array of expressions into a string, (join [Position] ************************ 's the mate you're looking for!joinWill also stringify anything it meets. Without separator: (**********************************************


>cat test.json | jql '("countries" ((keys) (join (array ("name") ("population") ("european")))))' [  "Poland3.8e 07true",  "United States3.27e 08false",  "Germany8.3e 07true"]With separator:  (**********************************************
>cat test.json | jql '("countries" ((keys)) join (array ("name") ("population") ("european")) ",")))' [  "Poland population: 38000000",  "United States population: 327000000",  "Germany population: 83000000"]




(************************************************************ (******************************************** (********************************************** (sprintf) Whenever I learn a new language, I feel much more comfortable when I know there's a sprintf function. (and how to use it)

















Anyways, here you go, the syntax is the same as that of the go standard library (fmt.Sprintf) ********************************************* (function: (**********************************************


>cat test.json | jql '("countries" ((keys) (sprintf "% s population:% .0f" ("name") ("population"))))' [  "Poland population: 38000000",  "United States population: 327000000",  "Germany population: 83000000"]Hope you're feeling comfortable[  "Poland3.8e 07true",  "United States3.27e 08false",  "Germany8.3e 07true"]  now:)
Logic  (******************************** Yey, back to the basics!
You've got  (eq) ,  (lt) ************************************************** and  (gt) **************************************************, all working as you'd probably expect:
>cat test.json | jql '(eq "test" "test")' true>cat test.json | jql '(eq "test" "test2")' false>cat test.json | jql '(lt "a" "b")' true>cat test.json | jql '(lt "b" "a")' false>cat test.json | jql '(gt 5 4)' trueIn case you're wondering,


eqdoes a  reflect. DeepEqual [Array[Int] on both arguments.


You've also gotand,  (or) ,  (not) ****************************************************, to cover your back when tackling those primal and primitive (some would say  (fundamental)  problems you may encounter:




>cat test.json | jql '(and true true true)' true>cat test.json | jql '(and true true false)' false>cat test.json | jql '(and true true null)' false>cat test.json | jql '(or true true false)' true>cat test.json | jql '(or)' false>cat test.json | jql '(and)' true>cat test.json | jql '(not true)' false>cat test.json | jql '(not false)' true>cat test.json | jql '(not null)' true>cat test.json | jql '(not (array false))' false(****************************************** and




andorare both lazy.






(**********************************************************  (Truthiness)   This brings us to the topic of truthiness. What does


and




consider to be "true"? Well, it's quite simple actually.


(**************************************************************************************** (null) ******************************************* (is not) ****************************************** (truthy.)   (false) ***************************************** (is nottruthy.  (************************************************************************************** (anything else) ****************************************** istruthy.  (**************************************************************************************  (****************************************************************************************
ifte sounds kinda fluffy, but unfortunately it only stands for If Then Else.


>cat test.json | jql '(ifte true "true" "false")' "true"It’s too lazy. If it weren't, this would error:
l>cat test.json | jql '(ifte true "true" (error ":("))' "true"Fluffy and lazy. Like a cat. Who doesn't like cats? Who doesn't like ifte?  (🐈) *******************************************************************




Sometimes you want just part of the cake, the part with no.


I've got no data set on cakes though, so let's get back to our beloved countries:




>cat test.json | jql '("countries" (filter (gt ("population") [Array[Int] ))) ' [eu_since:2004 european:true name:Poland population:3.8e 07]Also, because  null


is not truthy, and  (elem) **************************************************** (returns) ************************************************** (false) **************************************************** if it encounters missing fields or indices, you can use  (filter)  to get rid of wrong-schema data:


>cat test.json | jql '("countries" (filter ("eu_since")))' [  {    "eu_since": "2004",    "european": true,    "name": "Poland",    "population": 38000000  },  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }](******************************************************************************************  (**********************************************************






(******************************************** (pipe) **********************************************************************  (****************************************** (pipe)  It is a fairly useless function because you can just use a bash pipe. But if for some reason you want to save cpu cycles:


>cat test.json | jql '(pipe                            ("countries")                            ((range 2))                            ((keys) ("name"))) ' [  {    "european": false,    "name": "United States",    "population": 327000000  },  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }]is equal to  (**********************************************
>cat test.json | jql '("countries")' | jql '((range 2))' | jql '((keys) ("name"))' [  {    "european": false,    "name": "United States",    "population": 327000000  },  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }](******************************************************************************************  (********************************************************** (********************************** (recover) ********************************************************************  Finally, the one whose name shall not be spoken out loud.  (👹)  Needing him means you either encountered a bug in  (jql) *******************************************, or that your dataset is seriously botched.
He'll help you catch errors and panics, nullyfing the expression, leaving only void behind.

	

			

	

			

		
	

		







>cat test.json | jql '("countries" ((keys) (rec *** (ifte ("european") (id) (error "not european")))))' [  {    "eu_since": "2004",    "european": true,    "name": "Poland",    "population": 38000000  },  null,  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }]Combine him with  (filter)  and even the void will be gone!
In practice you obviously have to spell out his name, otherwise it won't work, but that's on you!
Hope you enjoyed this  (incredible) ****************************************** (journey!)
Moreover, I hope it's not the end of it! Hordes of JSON blobs still await and I hope
jql
will become your weapon of choice for dealing with them from now on![Int]
Issues,  (⭐️) ***************************************************************** stars  (⭐️) ****************************************************************, comments, messages, reviews, benchmarks, you name it! - all are very appreciated!  (😉) ****************************************************************
 (************************************************ JSON: Any value Expression [Position]: (JSON ->T) Expression can be seen as a Continuation elem:     With one arg: (Expression [Position]) ->(Expression [    "United States",    327000000  ]=(elem position (id))     With two args: (Expression [  {    "eu_since": "2004",    "european": true,    "name": "Poland",    "population": 38000000  },  null,  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }] x Expression [T]) ->(Expression [eu_since:2004 european:true name:Poland population:3.8e 07]) keys: () ->(Expression [T]) id: () ->(Expression [Position] array: (Expression [  {    "eu_since": "2004",    "european": true,    "name": "Poland",    "population": 38000000  },  null,  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }] ...) ->(Expression [Array [Position]) (T's can vary) object: ((Expression x Expression [  "Poland",  "United States"]) ...) ->(Expression [  {    "eu_since": "2004",    "european": true,    "name": "Poland",    "population": 38000000  },  null,  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }] pipe: (Expression ...) ->(Expression) sprintf: (Expression [Array[T] x Expression  ...) ->(Expression [Array[T] ) join: (Expression [  {    "eu_since": "2004",    "european": true,    "name": "Poland",    "population": 38000000  },  null,  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }] ...) ->(Expression [    "United States",    327000000  ]  filter: (Expression [Array[T]) ->(Expression [Position] eq, lt, gt: (Expression x Expression) ->(Expression [Array[T]) range:     With one arg: (Expression [String]) ->(Expression [Array [Int]])     With two args: (Expression [String] x Expression [Bool] ->(Expression [Array [european:false name:United States population:3.27e 08] **************]) and, or: (Expression [Array[T] ...) ->(Expression [String] not: (Expression [Bool]) ->(Expression [  "United States",  "Germany"] ) ifte: (Expression [Array[T] x Expression  (x Expression) ) ->(Expression [B] ) error: (Expression [JSON]) ->(!) recover: (Expression [T]) ->(Expression [    "Poland",    "United States",    "Germany"  ] )
This is a fairly synthetic benchmark ( It's the first that came to my mind), so I won't say  (jql)  is faster than jq. Especially since jq has various much more advanced functionalities. What I will say though, is that  jql  is definitely not slow and you can freely cut your way through gigabytes of data quickly, as in the following benchmarks it was 2-3 times faster than jq.
The benchbig.json file contains 2 ^ 23 repetitions of the json we've been using in the examples so far, and the benchmedium.json file contains 20
(Command)Mean [B] ******************************************************************************************************  (Min) ************************** Min **************************************************************************************************  (Max) ********************************************************************************************************************************  (Relative)  (******************************************************************************************  (**********************************************************************************************************
cat benchbig.json | jql '("countries" ((keys) ("name")))'>>out.json  [  {    "european": false,    "name": "United States",    "population": 327000000  },  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }] ************************************************************************************  (5) *************************************************************************************************************************************************** (± 0.) ************************************************************************************************************************************************************************************************************  (*********************************************************************************************************** (5) ******************************************************************************************************************************************************  (*********************************************************************************************************** (6) ****************************************************************************************************************************************************************************************************************************************
91 ± 53.


(******************************************** cat benchbig.json | jq '.countries (**************************************. name'>>out. json
(****************************************************************************************************************************************************************************************************************************. ************************************************************************************************************************************************ (± 0.) ********************************************************************************************************************************************************************************************************(****************************************************************************************************************************************************************************************************************************. **************************************************************************************************************************************************(**************************************************************************************************************************************************************************************************************************. ****************************************************************************************************************************************************************************************************2019.  [  "Poland population: 38000000",  "United States population: 327000000",  "Germany population: 83000000"] **********************************************************************************************************************************************. **************************************************************************************************************************************************************************************







(******************************************** cat benchmedium.json | jql '("countries" ((keys) ("name")))'>>out.json  [  {    "european": false,    "name": "United States",    "population": 327000000  },  {    "eu_since": "1993",    "european": true,    "name": "Germany",    "population": 83000000  }] ************************************************************************************  (0.0) ********************************************************************************************************************************************************************************************************************************** (± 0.) ******************************************************************************************************************************************************************************************************************************************* (0) ************************************************************************************************************  (0.0) ******************************************************************************************************************************************************************************************************************************** (0.0) ******************************************************************************************************************************************************************************************************************************  (*********************************************************************************************************** (1.) ******************************************************************************************************************************************************************************************************************************************



(******************************************** cat benchmedium.json | jq '.countries (**************************************. name'>>out. json
(0.0) ****************************************************************************************************************************************************************************************************************** (± 0.) ******************************************************************************************************************************************************************************************************************************************  (*********************************************************************************************************** (0.0) ********************************************************************************************************************************************************************************************************************  (*********************************************************************************************************** (0.) ************************************************************************************************************************************************************************************************************  (*********************************************************************************************************** (3) ******************************************************************************************************************************************************************************************************************** (± 0.) ************************************************************************************************************************************************************************************************************ (**************************************** The benchmarks were run using  (hyperfine) ************************************** on a preheated (very loud) Macbook Pro  (mid -) ***************************************************************************************************************************************** (i7 2.8GHz) ************************************************************************************************************************************************************************************************************************* (GB) ******************************************************************************************************************************************************************* (GB.)
You can generate the benchmark data with benchmarks / gen.sh.




   (****************************************************************************************************************Brave Browser(************************************************************************************************************** (Read More) *********************************************
327000000

What do you think?

0 Points
Upvote Downvote

Leave a Reply

Your email address will not be published. Required fields are marked *

GIPHY App Key not set. Please check settings

Bitcoin Blows Past $ 8,100 as New Accumulation Phase Begins, Crypto Coins News

Bitcoin Blows Past $ 8,100 as New Accumulation Phase Begins, Crypto Coins News
François Chollet on Twitter, Hacker News

François Chollet on Twitter, Hacker News