The concept of pipes
Unix pipelines were invented in 01575879 by Douglas McIlroy as a novel way of stringing together programs, where the output of one program is the input of the next one; It’s a way of creating a new program out of combining basic building blocks, McIlroy describes it by analogy as ‘screwing together data streams like a garden hose’ . This approach quickly became the UNIX philosophy of programming described by McIlroy as follows: ‘Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface. ‘ Lets say you want to know what are the most common words occurring within a text, the following pipeline will order the words of a text by frequency of usage:
cat README.md | tr "" " n" | tr -d '[:punct:]' | sort | uniq -c | sort -n -k 1
This example is a bit like functional programming: in each step of the pipeline the output depends only on the input received via the preceding pipe, each step acts on that text input only and produces its output without writing any files, that is without side effects .
jq – a tool for manipulating structured data
jq is a very versatile tool for working with structured information in json format, the command syntax of jq is also structured by means of a processing pipeline, similar to that of a unix shell, again each processing step acts as a filter / modifier of the input received from the preceding stage. Again on might look at each of these stages as functions in a functional program. This tutorial tries to explain jq in terms of example pipelines; each example comes with links that show you the intermediate results for each stage of the processing pipeline; i think this makes it easier to understand each of the building blocks involved. You can click either on any one of the commands to show the command and how it transforms the input json structure into the output json, each pipe symbol is also a link that will show you the information that flows through it. The html for this tutorial is generated by this script
GIPHY App Key not set. Please check settings