One thing that I love about bash is that there is never a shortage of new tips and tricks to learn. I have been using bash for over 10 years now and just stumbled on this little trick.
This one (as the title implies) allows you to quickly substitute a string into the previous command and rerun the command with the substitution.
Quick substitution is officially part of the Bash Event Designators mechanism and is a great way to fix a typo from a previous command. Below is an example.
# Simple example to highlight substitutions
echo foo
# This will replace the string "foo" with "bar" and rerun the last command
^foo^bar
This shorthand notation is great for most use cases, with the exception of needing to replace multiple instances of a given string. Luckily that is easily addressed with the Event Designators expanded substitution syntax, shown below.
# This will substitute ALL occurrences of foo in the previous command
!!:gs/foo/bar/
# Slightly different syntax allows you to do the same thing in ZSH
^foo^bar^:G
The syntax is slightly more complicated in the first example but should be familiar enough to anyone that has used sed and/or vim substitutions, and the second example is almost identical to the shorthand substitution.
fc
Taking things one step further, we can actually edit the previous command to fix anything more than a typo of different argument. fc is actually a bash builtin function so it is available almost everywhere.
fc is especially useful for dealing with very long, complicated commands.
# Oops, we messed this up
echo fobarr | grep bar
# To fix it, just open the above in your default editor
fc
# when you write and quit the file it will put the contents into your current command
There are many great tutorials available so I would recommend looking around to see all the options and get more ideas.
The landscape of command line driver text manipulation and processing tools is somewhat large and confusing, with more and more tools emerging all the time. Because I am having trouble keeping them all in my head, I decided to make a little reference guide to help remember which tool to choose for the correct task at hand.
Reach for jq first when you need to do any kind of processing with JSON files.
From the website, “jq is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.”
sudo apt-get install jq
# or
brew install jq
# consume json and output as unchanged json
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'
This project looks very interesting and is a refreshing thing to see in the Linux world. JC is basically a way to create structured objects (ala PowerShell) as JSON output from running various Linux commands. And from the GitHub repo, ” This tool serializes the output of popular gnu linux command line tools and file types to structured JSON output. This allows piping of output to tools like jq”.
Transforming data into structured objects can massively simplify interacting with them by pairing the output with jq to interact with.
This tool can be confusing because there is both a Python version and a Go version. On top of that, the Python version includes its own version of xq, which is different than the standalone xq tool.
The main differences between the Python and Go version is that the Python version can deal with both yaml and xml while the Go version is meant to be used as a command line tool to deal with only yaml.
From the website, oq is “A performant, portable jq wrapper thats facilitates the consumption and output of formats other than JSON; using jq filters to transform the data”.
The claim to fame that oq has is that it is very similar to jq but works better with other data formats, including xml and yaml. For example, you can read in some xml (and others), apply some filters, and output to yaml (and others). This flexibility makes oq a good option if you need to deal with different data formats oustide of JSON.
snap install oq
# or
brew tap blacksmoke16/tap && brew install oq
# consume json and output xml
echo '{"name": "Jim"}' | oq -o xml .
From the Github page, “Apply XPath expressions to XML, like jq does for JSONPath and JSON”.
The coolest use case I have found for xq so far is taking in an xml file and outputting it into a json file, which surprising I haven’t found another tool that can do this (oc authors say there are plans to do this in the future). The simplest example is to curl a page, pipe it through xq to change it to json and then pipe it again and use jq to manipulate the data.
This is a newer tool, with a slightly different approach aimed at helping to automate configurations, especially for things like Kubernetes but should work with most structured data.
This tool works with json, yaml and hcl and can be used in conjunction with Javascript, making it an interesting option.
curl -Lo jk https://github.com/jkcfg/jk/releases/download/0.3.1/jk-darwin-amd64
chmod +x jk
sudo mv jk /usr/local/bin/
// alice.js
const alice = {
name: 'Alice',
beverage: 'Club-Mate',
monitors: 2,
languages: [
'python',
'haskell',
'c++',
'68k assembly', // Alice is cool like that!
],
};
// Instruct to write the alice object as a YAML file.
export default [
{ value: alice, file: `developers/${alice.name.toLowerCase()}.yaml` },
];
jk generate -v alice.js
This is basically a simplified templating language that only intends to deal with yaml. The approach the authors took was to create yaml templates and sanbdox/embed Python into the templating engine, allowing users to call on the power of Python inside of their templates.
The easiest way to play around with ytt if you don’t want to clone the repo is to try out the online playground.
From the GitHub page, ” Configula is a configuration generation language and processor. It’s goal is to make the programmatic definition of declarative configuration easy and intuitive”.
Similar in some ways to ytt, but instead of embedding Python into the yaml template file, you create a .py file and then render the py file into yaml using the Configula command line tool.
git clone https://github.com/brendandburns/configula
cd configula
# tiny.py
# Define a YAML object where the 'foo' field has the value of evaluating 1 + 2 (e.g. 3)
my_obj = foo: !~ 1 + 2
my_obj.render()
./configula examples/tiny.py
Jsonnet bills itself as a “data templating language for app and tool developers”. This tool was originally created by folks working at Google, and has been around for quite some time now. For some reason always seems to fly underneath the radar but it is super powerful.
This tool is a superset of JSON and allows you to add conditionals, loops and other functions available as part of its standard library. Jsonnet can render itself into json and yaml output.
For (pretty much) everything else, there is Sed. Sed, short for stream editor, has been around forever and is basically a Swiss army knife for manipulating text, and if you have been using *nix for any length of time you have more than likely come across this tool before. From their docs, Sed is “a stream editor is used to perform basic text transformations on an input stream”.
The odds are good that Sed will likely do what you’re looking for if you can’t use one of the aforementioned tools.
