Introduction

My personal notes. The notes divided into following blocks:

• Collections: my favorite command-line tools and utility websites.
• Languages: my learning notes on some programming languages.
• Tools: my experience with famous tools.

Collections

A collections of tools, services, websites.

• CLI Tools
• Utility Websites

CLI Tools

fzf

An interactive filter program for any kind of list.

rg

A better grep

jq

Command-line JSON processor

yq

Process YAML, JSON, XML, CSV and properties documents from the CLI

bat

Cat with syntax highlight

jless

A command-line JSON viewer

just

A command runner

argc

Easily create and use cli that based on bashscript.

rclone

Rsync for cloud storage

rsync

An open source utility that provides fast incremental file transfer

dufs

A file server that supports static serving, uploading, searching, accessing control, webdav…

mdbook

Create book from markdown files.

zellij

A terminal multiplexer similar to tmux.

mcfly

Search shell history

croc

Easily and securely send things from one computer to another with cli

frp

A proxy tools supports intranet penetration, port forwarding.

proxychains-ng

Run command with proxy network

webhookd

Use shell scripts to handle webhook request.

yt-dlp

A feature-rich command-line audio/video downloader

Utility Websites

amp-what

Discover Unicode Character Entities & Symbols.

webhook.site

Test, process and transform HTTP requests.

draw.io

Draw wireframes, support local storage, cloud storage.

asciiflow

Draw ascii diagrams.

iconfont.cn

Pick icons to create your own icon fonts.

squoosh

A image compression and format conversion tool.

Languages

Some notes on programing languages.

• Shell
• Rust
• Markdown
• Nodejs

Shell

Variables

# Default value
echo ${value:-xa} # xa
value=xyz/z-a/abc
# Head
echo ${value%/*} # xyz/z-a
echo ${value%%/*} # xyz
# Tail
echo ${value#*/} # z-a/abc
echo ${value##*/} # abc
# Truncate
echo ${value:7} # /abc
echo ${value:7:4} # /abc
echo ${value:(-4):4} # /abc
# Replace
echo ${value/a/A} # xyz/z-A/abc
echo ${value//a/A} # xyz/z-A/Abc

str="Hello World"
echo ${str,,}   # hello world (Lowercase)
echo ${str^^}   # HELLO WORLD (Uppercase)
echo ${str^}    # Hello World (Only first char)

Shell Variables

• $@: args list (recommended)
• $*: spconcatenate into a string (easy to mess up)
• $#: count args
• $n: nth arg
• $?: last exit code
• $!: last pid

Array

fruits=(Apple Banana Orange Pear)

echo "${fruits[0]}"           # Element #0
echo "${fruits[-1]}"          # Last element
echo "${fruits[@]}"           # All elements, space-separated
echo "${#fruits[@]}"          # Number of elements
echo "${#fruits}"             # String length of the 1st element
echo "${#fruits[3]}"          # String length of the Nth element
echo "${fruits[@]:1:2}"       # Range (from index 1 (second element), length 2)
echo "${!fruits[@]}"          # All keys: 0 1 ...

fruits=("${fruits[@]}" Grape)          # Push
fruits+=(Watermelon)                   # Also Push
unset fruits[2]                         # Remove one item
fruits=("${fruits[@]}")                 # Duplicate
fruits=("${fruits[@]}" "${Veggies[@]}") # Concatenate

for i in "${fruits[@]}"; do
  echo "$i"
done

if [[ " ${fruits[*]} " =~ " Banana " ]]; then
  echo "Banana in fruits"
fi

Dictionary

declare -A sounds

sounds[dog]="bark"
sounds[cow]="moo"
sounds[bird]="tweet"
sounds[wolf]="howl"

echo "${sounds[dog]}" # Dog's sound
echo "${sounds[@]}"   # All values
echo "${!sounds[@]}"  # All keys
echo "${#sounds[@]}"  # Number of elements
unset sounds[dog]     # Delete dog

# Value loop
for val in "${sounds[@]}"; do
  echo "$val"
done

# Key loop
for key in "${!sounds[@]}"; do
  echo "$key"
done

Condition

Test

Test equal with glob

• *: Any length of any character, e.g. [[ "$v" == *abc* ]]
• ?: any single character, e.g. [[ "$v" == rc? ]]
• [] any character in set, e.g. [[ "$v" == [a-z] ]]
• [!]: any character not in set, e.g. [[ "$v" == [!0-9] ]]

Test equal with extglob

shopt -s extglob

• ?(pattern)：0 or 1
• *(pattern)：0 or many
• +(pattern)：1 or many, e.g. [[ "$v" == +([0-9]) ]]
• @(pattern)：exact, e.g. [[ "$v" == @(apple|banana|orange) ]]
• !(pattern): exclude, e.g. [[ "$v" == !(tmp|cache) ]]

Use if expr

if [[ "foo" == "$1" ]]; then
  echo foo
elif [[ "bar" == "$1" ]]; then
  echo bar
else
  echo "not found"
  exit 1
fi

if grep -q 'alice' /etc/passwd; then
  echo "User alice exists"
fi

read -r -p "Are you sure (y/n)? " choice
if [ "$choice" = "y" ]; then
  echo choice yes
fi

Use case expr

case "$1" in
  start | up)
    echo start
    ;;
  stop | down)
    echo stop
    ;;

  *)
    echo "Usage: $0 {start|stop}"
    ;;
esac

Loop

# index
for i in {1..10}; do
  echo "$i"
done

for ((i=1; i<=10; i+=2)); do
  echo "$i"
done

# fs
for f in ./*; do
  echo "$f"
done

# array
IFS=, read -r -a arr <<< "a,b,c"
for v in "${arr[@]}"; do
  echo "$v"
done

# lines
while IFS= read -r l; do
  echo "$l"
done < file.txt

# forever
while true; do
  sleep 1
done

Function

Shell function has no real return value

• return can only return an exit code (0-255)
• echo is the actual way to return data

foo() {
    local msg='world'
    echo "$1 $msg"
}
result="$(foo hello)"
echo "$result"

_is_win() {
    [[ "$(uname)" =~ "_NT" ]]
}

if _is_win; then
    echo 'windows'
fi

Math

$((1 + 2))
$((10 - 3))
$((4 * 5))
$((20 / 3))   # Integer division => 6
$((RANDOM % 100))  # Random number 0..99

$((a = 5 + 3))
$((a++))
$((a--))
$((a += 10))
$((a *= 2))

((a == b))
((a != b))
((a < b))
((a <= b))
((a > b))
((a >= b))

Redirect

• <: stdin
• >: stdout
• >>: stdout in append mode
• 2>: stderr
• 1>&2： redirect stdout to stderr
• 2>&1： redirect stderr to stdout
• >file 2>&1: redirect stdout/stderr to file

Here String

grep foo <<<"$HOME"

Here Document

cat << EOF | sudo tee -a /etc/sudoers.d/$USER
$USER ALL=(ALL) NOPASSWD:ALL
EOF 

• <<EOF: will replace variable
• <<'EOF': not replace variable
• <<-EOF: will replace variable trim \t only
• <<-'EOF': not replace variable，trim \t only

Process Substitution

• <(cmd)：Use command stdout as a file, e.g. diff <(sort a.txt) <(sort b.txt)
• >(cmd): Write to the command stdin, e.g. tar cf >(gzip > out.tar.gz) dir/

Subshell

Modification of internal variables does not affect external ones

(cd /tmp && touch test.txt)

Command group

Executes in the current shell, used only for logical grouping

{ echo "header"; cat data.txt; echo "footer"; } > output.txt

There must be a space after {, and a semicolon or newline before }

Set Options

Misc

• source: Used for Running script files in the current Shell environment

source file.sh
. file.sh

• trap: Used to capture signals (such as EXIT, INT, TERM) and automatically execute specified cleanup commands when the script exits or is interrupted.

cleanup() {
    rm -f "$tmpfile"
    echo "Cleaned up on exit"
}
trap cleanup EXIT       # Execute when the script exits (including normal and error exits)
trap 'echo Ctrl+C' INT  # Capture SIGINT

• Export all environments/variables/functions/options of the current shell

( set -o posix; set )

• Load dotenv file

[ -f .env ] && { set -a; source .env; set +a; }

Common Tools

• Character related: find/sed/awk
• Network related: curl/wget
• Argument parser related: getopt/argc
• JSON/YAML/TOML related: jq/yq
• Interative related: gum

Rust

Installation

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Rustup

Concepts

• target: x86_64-pc-windows-msvc, x86_64-unknown-linux-gnu, wasm32-unknown-unknown
• toolchain: stable-x86_64-pc-windows-msvc, nightly-2021-05-11-x86_64-unknown-linux-gnu
• component: rustc,rustsrc,rustfmt, rust-analyzer

Snippets

rustup update 

rustup toolchain install nightly

rustup target add wasm32-unknown-unknown --toolchain nightly

rustup default nightly

rustup override set nightly

Crates

• tokio: Async
• crossbeam: Concurrency
• rayon: Concurrency computation
• axum: web framework
• sea-orm: orm
• sqlx: sql
• reqwest: http client
• anyhow: generic error framework
• thiserror: library error framework
• clap: cli framework
• nom: bytes/charactor parser
• assert_cmd: cli test framework
• insta: snapshot test framework

Cargo

Plugins

• cargo-outdated: Displaying when dependencies have newer versions available.
• cargo-audit: Audit your dependencies for crates with security vulnerabilities.
• cargo-udeps: Find unused dependencies in Cargo.toml
• cargo-bloat: Find out what takes most of the space in your executable.
• cargo-fuzz: Fuzzing with libFuzzer
• cargo-insta: Snapshot testing
• cargo-binstall: Binary installation for rust projects
• cargo-geiger: Detects usage of unsafe Rust in a Rust crate and its dependencies.

Snippets

cargo add serde
cargo add serde -F derive
cargo add -D serde
cargo add -B serde

cargo tree
cargo tree -p libc
cargo tree -p libc -i

Cargo.toml

[profile.release]
lto = true
strip = true
opt-level = "z"

Resources

• Rust Language Cheat Sheet
• Rust by Example
• Rust Design Patterns
• Rust Atomics and Locks
• learn project: toydb
• learn project: mini-redis
• Rust API Guidelines
• pretzelhammer Blog
• Jon Gjengset Youtube

Markdown

Heading

Heading3

Heading4

Heading5

Text Style

bold text

italicized text

blockquote

Ordered List

1. First item
2. Second item
3. Third item

Unordered List

• First item
• Second item
• Third item

Code

Inline code with backticks

#![allow(unused)]
fn main() {
#[derive(Debug)]
pub enum State {
    Start,
    Transient,
    Closed,
}

impl From<&'a str> for State {
    fn from(s: &'a str) -> Self {
        match s {
            "start" => State::Start,
            "closed" => State::Closed,
            _ => unreachable!(),
        }
    }
}
}

Link

Link

Image

Horizontal Rule

Autolink literals

www.example.com, https://example.com, and contact@example.com.

Footnote

A note¹

Strikethrough

one or two tildes.

Table

Tasklist

• to do
• done

Math

$E = mc^2 + 2$

1. Big note. ↩

Nodejs

Installation

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash

nvm i --lts

nvm i --lts --reinstall-packages-from=node

NPM

# list all packages
npm ls -g --depth=0
# init projects
npm i -y

Run node service

1. Use pm2

pm2 start --name app npm start

2. Use systemd

[Unit]
After=network.target

[Service]
Environment=NODE_PORT=3000
Type=simple
User=ubuntu
ExecStart=/usr/bin/node /home/ubuntu/app.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo cp app.service /etc/systemd/system/
sudo systemd daemon-reload
sudo systemd start app

3. Use docker

FROM sigoden/node:native as builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --prod

FROM sigoden/node:slim
WORKDIR /app
COPY --from=builder /app .
COPY . .
EXPOSE 3000
CMD ["node", "index.js"]

Native addon

1. INstall deps like gcc, make, python

# ubuntu
apt install build-essential

# alpine
apk add make gcc g++ python3 git

2. Test native addon

npm i --build-from-source bcrypt

ES features

https://node.green

Tools

Some notes on tools.

• Docker
• Git
• Github CI
• Linux
• Redis
• Vim

Docker

Installation

curl -fsSL get.docker.com | sudo bash

None-Root

sudo usermod -aG docker $USER

Snippets

# Run container then enter shell
docker run -it --rm -v `pwd`/data:/data -p 3000:3000 node:16 bash

# Run container as service
docker run -d --name redis -p 6379:6379 redis:5

# Exec in container
docker exec -it redis bash
docker exec redis bash -c 'redis-cli set k1 a && redis-cli get k1'

# View log
docker logs --tail 50 -f redis

# Clean
docker system prune -f

# Clean dead images
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

Use Mirror

• WIN: Docker Desktop -> Preferences -> Docker Engine
• LINUX: /etc/docker/daemon.json

{
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"]
}

Dockerfile

COPY vs ADD

• ADD copy files, download remote resources, handle zip/tar.gz
• COPY only copy local files

Do not use ADD unless COPY not works.

ADD /source/file/path  /destination/path
ADD http://source.file/url  /destination/path
ADD source.file.tar.gz /destination

COPY /source/file/path  /destination/path

ENTRYPOINT vs CMD

• No ENTRYPOINT

• ENTRYPOINT e1

• ENTRYPOINT [“e1”]

Conconcate commands

RUN set -ex \
    # comment
    && apt-get update && apt-get install -y curl --no-install-recommends \
    && rm -rf /var/lib/apt/lists/* 

Layers

FROM sigoden/node:native as builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --prod

FROM sigoden/node:slim
WORKDIR /app
COPY --from=builder /app .
COPY . .
EXPOSE 3000
CMD ["node", "index.js"]

Heredoc

RUN printf '#!/bin/bash\n\
echo 1\n\
echo 2'\
>> /tmp/hello

Enrypoint file

#!/bin/sh
set -e

if [ "${1#-}" != "${1}" ] || [ -z "$(command -v "${1}")" ]; then
  set -- app "$@" # change app to your binary
fi

exec "$@"

Normal User

RUN groupadd --gid 1000 user \
  && useradd --uid 1000 --gid user --shell /bin/bash --create-home user

Multi-Arch Build

docker buildx create --name mybuilder --driver docker-container
docker buildx use mybuilder

docker buildx build \
  --platform linux/amd64,linux/arm64,linux/arm/v7,linux/386 \
  -t demo . 

Git

Stages

Clone

# branch
git clone -b dev https://github.com/org/repo.git
# target folder
git clone https://github.com/org/repo.git myrepo 
# contains submodule
git clone --recurse-submodules https://github.com/org/repo.git
# abaddon history
git clone --depth=1 https://github.com/org/repo.git

Branch

# list branches
git branch
# list remote branches
git branch -r
# list all branches
git branch -a
# create branch
git checkout -b feat1
# rename a branch
git branch -m newname
# checkout branch
git checkout feat1
# checkout remote branch
git checkout -t origin/dev
# delete branch
git branch -d feat1
# delete branch forcedly
git branch -D feat1
# delete branch of remote repo
git push origin :feat1

Tag

# list tags
git tag
# create tag
git tag v1.0.0
# delete tag
git tag -d v1.0.0
# delete tag forcedly
git tag -D v1.0.0
# delete tag of remote repo
git push origin :v1.0.0

Fetch

# pull changes
git fetch
# pull github pr
git fetch origin pull/ID/head:BRANCH_NAME
# pull changes and prune none-exist remote branches
git fetch --purge

Snippets

# merge last commit
git commit --amend
# sync submodule
git submodule update --init --recursive
# inspect remote url
git remote -v
# change remote url
git remote set-url origin $new_repo

Config

User

git config --global user.name $user
git config --global user.email $email

Ignore

# linux/macos
git config --global core.excludesFile '~/.gitignore'
# windows
git config --global core.excludesFile "$env:USERPROFILE\.gitignore"

Cjk path

git config --global core.quotepath false

CRLF

git config --global core.eol lf
git config --global core.autocrlf input

Credentials

WSL2

git config --global credential.helper \
  "/mnt/c/Program\\ Files/Git/mingw64/bin/git-credential-manager.exe"

SSH Repo

sudo useradd -m -r -U -d /home/git-repos git
sudo su - git

mkdir -p ~/.ssh
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
touch ~/.ssh/authorized_keys

git init --bare ~/repo.git

git remote add origin git@<your-server-ip>:repo.git

Github CI

Triggers

on: 
  push:
    tags: 
      - v[0-9]+.[0-9]+.[0-9]+*

on
  push:
    branches:
    - main
  pull_request:

on:
  workflow_dispatch:
  schedule:
    - cron:  '0 2 * * *'

Environment Variables

jobs:
  job1:
    runs-on: ubuntu-latest
    env:
      FOO: foo
    steps:
      - if: ${{ env.FOO == 'foo' }}
        run: echo "$BAR"
        env:
          BAR: bar

Debug Tips

Print all environment variables

    steps:
      - run: env

Print github variables

    steps:
      - run: echo "$GITHUB_CONTEXT"
        env:
          GITHUB_CONTEXT: ${{ toJSON(github) }}

Print shell evalutaion

    steps:
      - run: |
          set -x 

          src=`pwd`
          dist=$src/dist
        shell: bash

ssh into runner

    steps:
    - name: Setup upterm session
      uses: lhotari/action-upterm@v1

RUNNER will log:

=== S1WV8GFLUQTD7SN7TAKM                                                                 
Command:                tmux new -s upterm -x 132 -y 43                                 
Force Command:          tmux attach -t upterm                                           
Host:                   ssh://uptermd.upterm.dev:22                                     
SSH Session:            ssh s1wv8gfLuqTd7sn7taKm:MTAuMjQ0LjAuNzY6MjI=@uptermd.upterm.dev

Get Tag info

jobs:
  release:
    outputs:
      rc: ${{ steps.check-tag.outputs.rc }}
    steps:
      - name: Get Tag
        id: get_tag
        shell: bash
        run: |
          tag=${GITHUB_REF##*/}
          if [[ "$tag" =~ [0-9]+.[0-9]+.[0-9]+$ ]]; then
            echo "rc=false" >> $GITHUB_OUTPUT
          else
            echo "rc=true" >> $GITHUB_OUTPUT
          fi

Add environment variable

jobs:
  job1:
    steps:
      - name: Set env var
        run: |
          echo "foo=bar" >> $GITHUB_ENV

Pass variables between jobs

jobs:
  job1:
    runs-on: ubuntu-20.04
    outputs:
      tag: ${{ steps.get_tag.outputs.tag }}

  job2:
    needs: ["job1"]
    if: needs.job1.outputs.tag != 'null'

Pass artificats between jobs

jobs:
  job1:
    steps:
      - uses: actions/upload-artifact@v4
        with:
          name: app
          path: target/release/app

  job2:
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: app
          path: ${{ github.workspace }}/target/release/app 

Build and push docker image

jobs:
  docker:
    name: Publish to Docker Hub
    if: startsWith(github.ref, 'refs/tags/')
    runs-on: ubuntu-latest
    needs: release
    steps:
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          build-args: |
            REPO=${{ github.repository }}
            VER=${{ github.ref_name }}
          platforms: |
            linux/amd64
            linux/arm64
            linux/386
            linux/arm/v7
          push: ${{ needs.release.outputs.rc == 'false' }}
          tags: ${{ github.repository }}:latest, ${{ github.repository }}:${{ github.ref_name }}

Linux

Inspect

File

# List files
ls -alh file

# Modify user group
chown -R alice file
chown -R alice:www-data file

# Modify perms
chmod 600 file
chmod -w file
chmod +x file
chmod o-x file
chmod u+w file

# Open with default app
xdg-open file

# link file
ln -s /user/bin/vim /usr/bin/vi
# Link file relative
ln -sr node_modules/eslint/bin/eslint.js node_modules/.bin/eslint

# Zip
zip -r data.zip data
# Inspect zip files
unzip -t data.zip
# Unzip file
unzip data.zip
# Unzip to specific folder
unzip -d output-dir data.zip
# Unzip with pass
zip -er data.zip data

# Tar
tar -cf data.tar.gz data
# Inspect tar files
tar -tf data.tar.gz
# Untar
tar -xf data.tar.gz
# Untar to specific folders
tar -xf data.tar.gz -C output-dir

# bz format compression (slower but smaller)
tar -cjf data.tar.bz2 data

# Download
wget $url
wget -O /tmp/file $curl
curl -o /tmp/file $curl

# Find
find -name README.md
find / -name sfz
find -type -f -name '*.log' -exec rm -rf {} \;

# split by number of copies
split -d -n 5 data.zip data.zip.part
# split by size
split -d -b 256M data.zip data.zip.part
# Combine split files
cat data.zip.part* > data.zip

# generate file
dd if=/dev/zero of=/tmp/1G.bin bs=1M count=1000
dd if=/dev/urandom of=/tmp/1G.bin bs=10M count=1000

User/Group

# add users (more complete, similar to interface operations)
adduser alice
# add user (recommended version)
useradd -m alice
# add user, specify user directory
useradd -d /d/alice alice
# add user, specify uid, gid
useradd -u 1001 -g 1001 nodejs
# add program account
useradd -r -s /usr/sbin/nologin nodejs

# MODIFY THE USER SHELL
usermod -s /usr/bin/bash alice

# user join group
usermod -a -G docker alice

# modify user password
passwd alice

# delete users
userdel nodejs

# add group
groupadd -g 344 linuxde
# delete group
groupdel linuxde

Process

# find process
ps aux | grep nginx

# kill process
kill $pid
# kill process forcelly
kill -9 $pid
# kill process by name
pkill nginx
pkill '^ssh$'
pkill -9 nginx

# find process by port
lsof -i:3000
netstat -utlpn | grep 3000
# find port by process
netstat -utlpn | grep nginx

cat /proc/$pid/status | grep VmRSS
cat /proc/$pid/cmdline 
cat /proc/$pid/environ 

ls -l /proc/$pid/cwd
ls -l /proc/$pid/exe
ls -l /proc/$pid/root

# run in backgroud
nohup server &
nohup server > server.log 2>&1 &
nohup server > server.log 2> server.err &

Text

# count lines
cat file | wc -l

# first n lines
cat file | head -3
# last n lists
cat file | tail -3

cat file | more

# hex
cat file | xxd 
cat file | xxd -p > file.hex

# split columns with `:`, only show first, third columns
cat /etc/group | cut -d: -f1,3
# only show 5-10 charactors of each row
cat file | cut -c5-10
# only show first 10 charactors of each row
cat file | cut -c-10
# only show 10th and subsequent of each line
cat file | cut -c10-

# replace charactor
cat file | tr '\t' ' '
# delete charactor
cat file | tr -d '\r'

# remove duplicates
cat file | sort | uniq 
# count duplicates
cat file | sort | uniq -c
# show duplicates
cat file | sort | uniq -d

sed

# delete first line
cat file | sed '1d'
# delete empty line
cat file | sed '/^$/d'
# insert at first line
cat file | sed '1i#!/usr/bin/env node\n'
# append on last line
cat file | sed '$a\\n'
# replace functions
cat file | sed '/foo() {/,/}/c foo() { }'
# replace path
cat file| sed 's|/bin/sh|/bin/bash|g'
# replace with matched parts
cat file | sed 's|\(\w\+\) \(\w\+\) \(\w\+\)|\2 \3 \1|'
# change file
sed -i 's/\w\+.ubuntu.com/mirrors.tuna.tsinghua.edu.cn/' /etc/apt/sources.list

awk

# print specific column
docker images | awk '{print $1}'
# specific delimeter
cat /etc/paswd | awk -F: '{print $1}'
# combine columns
cat file | awk '{print $1,$3}'

Regex

GLOB

Terminal

• !! last command
• !$ last postional parameter of last command

Set

set -euxo pipefail

• -e: Exit on none-zero status
• -x: Print a trace
• -u: Treat unset variables as error
• -o pipefail: Return none-zero if last pipeline return none-zero

Xargs

Capture the output of one command and pass it to another command

# Batch renames
ls -1 . | sed 's/\.mdx$//' | xargs -I{} mv {}.mdx {}.md

Cron Syntax

Min  Hour Day  Mon  Weekday
*    *    *    *    *  command to be executed
┬    ┬    ┬    ┬    ┬
│    │    │    │    └─  Weekday  (0=Sun .. 6=Sat)
│    │    │    └──────  Month    (1..12)
│    │    └───────────  Day      (1..31)
│    └────────────────  Hour     (0..23)
└─────────────────────  Minute   (0..59)

Common Snippets

Script Directory

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

Sudo Without Pass

cat << EOF | sudo tee -a /etc/sudoers.d/$USER
$USER ALL=(ALL) NOPASSWD:ALL
EOF 

Ulimit

cat <<EOF | sudo tee -a /etc/security/limits.d/$USER.conf
$USER soft nproc 100000
$USER hard nproc 100000
$USER soft nofile 100000
$USER hard nofile 100000
EOF

Change repo source

ubuntu

sed -i -E 's/\w+.ubuntu.com/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list

Alpine

sed -i 's/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apk/repositories

Editing Shortcuts

Redis

Clear all

EVAL "return redis.call('del', unpack(redis.call('keys', ARGV[1])))" 0 prefix:*

Enter redis container

#!/bin/bash -e
server=$(docker ps | grep redis | awk  '{print $1}')
docker exec -it $server redis-cli -a ${1:-pass}

vim

switch mode

move

search & replace

operate

examples：

text-object

例如：

normal mode

insert mode

visual mode

jump list / change list

mark

register

undo redo

macro

folding / z

window / tab

command mode

files

global

vimrc

syntax on
filetype plugin indent on

set nonumber
set linebreak
set showbreak=+++
set textwidth=100
set showmatch
set ruler
set visualbell
 
set hlsearch
set smartcase
set ignorecase
set incsearch
 
set autoindent
set expandtab
set shiftwidth=4
set softtabstop=4
set smarttab

set undolevels=1000
set backspace=indent,eol,start
if has('win32')
    set clipboard=unnamed
else
    set clipboard=unnamedplus
endif

JQ

jq is a stream-oriented transformation language. Filters consume an input and may produce zero, one, or many outputs

CLI Options

Input

-n

Use null as input — build data from scratch.

jq -n 'range(3)'

-R

Read input as raw strings (not JSON). Each line becomes a string.

printf 'foo\nbar\n' | jq -R '.'    # "foo" "bar"

-s

Slurp — read the entire input stream into a single array.

printf '{"x":1}\n{"x":2}\n' | jq -s 'map(.x) | add'    # 3

Output

-r

Raw output — print strings without JSON quotes/escaping.

$ jq -n '"hello"'
"hello"
$ jq -rn '"hello"'
hello

-b

Ensure no CRLF conversion on Windows.

$ jq -nb '42'

-c

Compact output — each output on a single line.

$ jq -n '[1,2]'
[
  1,
  2
]
$ jq -nc '[1,2]'
[1,2]

Variables

--arg

Set a variable $name to a string value.

jq -n --arg name foo '{$name}'    # {"name":"foo"}

--argjson

Set a variable $name to a JSON value.

jq -n --argjson n 42 '{$n}'    # {"n":42}

--args

Pass positional arguments as $ARGS.positional (strings).

jq -n '$ARGS.positional' --args hello world    # ["hello","world"]

--jsonargs

Pass positional arguments as JSON values.

jq -n '$ARGS.positional' --jsonargs 1 2 3    # [1,2,3]

Core language

Path Expr

.              Identity (root)
.foo           Object field access
.foo.bar       Nested object field access
."wired-key"   Key with special chars (quoted)
.foo?          Safe access (null if missing)
.[]            Iterate array elements
.[]?           Safe iterate overy arrays/objects
.[0]           Index by number (also .[0] = .[0] |= ...)
.[-1]          Last element (negative index)
.[1:5]         Array slice from index 1 (inclusive) to index 5 (exclusive)
.[1:]
.[:5]

Construction

Array & Object Construction

[1, 2, .a]       Build array
{key: .name}     Build object
{($k): .v}       Computed key
{name}           Shorthand for {name: .name}

Operator

|

Pipe the output of a filter as input to the next filter.

. | .name           Feed current value into field access
. | map(. + 1)      Feed array into map
keys | .[]          Chain filters — keys then iterate

The fundamental composition primitive: everything left of | is evaluated, and its output becomes . on the right.

jq -n '1 | . + 2'               # 3
jq -n '[1,2,3] | length'        # 3
jq -n '{a:1,b:2} | keys | .[]'  # "a" "b"

//

Alternative operator — returns left operand if it is not null or false, otherwise returns right operand. Useful for defaults.

.a // "default"      Use .a if present, else "default"
null // 42           → 42
false // 42          → 42 (false is falsy too)
"" // "fallback"     → "" (empty string is truthy)

?

? suppresses errors from a filter. Commonly paired with // for fallback defaults.

.foo?                 return null if key missing
.[]?                  return empty if value is not iterable
.foo?.bar? // "n/a"   chain safe accesses with default

jq -n '{} | .foo? // "no"'       # "no"
jq -n '5 | .[]? // "empty"'      # "empty"
jq -n 'null | .a?.b? // 0'       # 0

,

Produce multiple values from a single input as a stream. Each expression after , generates a separate output.

.name, .age        Output name and age as separate values
1, 2, 3            Stream of three numbers
[.x, .x * 2]       Build array from multiple expressions

jq -n '1, 2, 3'                 # 1 2 3 (three lines)
jq -n '{a:1,b:2} | .a, .b'      # 1 2
jq -nc '[1,2,3] | .[], .[]'     # 1 1 2 2 3 3

as

Bind the current value (or destructured parts) to a variable for later use.

. as $x | $x + 1
.a as $val | {$val}
{a, b} as {$x, $y} | $x + $y
.[] as $item | {item: $item}

Variables are prefixed with $ and stay in scope for the rest of the pipeline.

jq -n '1 as $x | $x + 2'            # 3
jq -n '[1,2,3] | . as $a | $a | add' # 6
jq -n '{a:1,b:2} | {a} as {$a} | $a' # 1

\(exp)

String interpolation

jq -rn '42 | "The input was \(.), which is one less than \(.+1)"'

Assignment

=        Assign
|=       Update value (e.g. .arr |= sort)
+= -= *= /= %=    Arithmetic update-assignment

jq -n '[1,2,3] | .[0] = 99'       # [99,2,3]
jq -n '{a:1} | .a |= . + 1'       # {"a":2}
jq -n '{a:1} | .a = (.a | . + 1)' # {"a":2}
jq -n '{a:1} | .a += 1'           # {"a":2}
jq -n '{a:10} | .a /= 2'          # {"a":5}

Arithmetic

.a + .b     Add (numbers) or concat (strings/arrays)
.a - .b     Subtract (or set diff: [1,2] - [2,3] → [1])
.a * .b     Multiply (or repeat string: "a" * 3 → "aaa")
.a / .b     Divide (or split string: "a,b" / "," → ["a","b"])
.a % .b     Modulo

jq -n '3 + 4 * 2'                 # 11  (* before +)
jq -n '[1,2] + [3,4]'             # [1,2,3,4]
jq -n '{"a":1} + {"b":2}'         # {"a":1,"b":2}
jq -n '{"a":1} * {"a":2,"b":3}'   # {"a":2,"b":3}  (merge, rhs wins)
jq -n '"a" * 3'                   # "aaa"
jq -n '"a,b" / ","'               # ["a","b"]

Comparison

.a == .b     Equal (type-sensitive, no coercion)
.a != .b     Not equal
.a > .b      Greater than
.a >= .b     Greater or equal
.a < .b      Less than
.a <= .b     Less or equal

Logic

.a > 1 and .b < 5       Both true
.a > 1 or  .b < 5       Either true
.flag | not             Negate via pipe (preferred idiom)
.a > 1 and (.b | not)   Group with parens

Standard library

Basic

length

Length of string, array, or object.

jq -n '"abc" | length'

keys / keys_unsorted

Return the keys of an object (sorted/insertion-order).

jq -n '{b:1,a:2} | keys'           # ["a","b"]
jq -n '{b:1,a:2} | keys_unsorted'  # ["b","a"]

to_entries / from_entries / with_entries

Convert between objects and arrays of key-value entries.

jq -n '{a:1} | to_entries'
jq -n '[{"key":"a","value":1}] | from_entries'
jq -n '{a:1,b:2} | with_entries(.value *= 2)'

Selection

numbers / strings / booleans / arrays / objects / nulls

Type filter functions — return only values of matching type.

jq -n '[1, "a", true, null] | map(numbers)'   # [1]
jq -n '[1, "a", true, null] | map(strings)'   # ["a"]

type

Return the type of a value as a string.

$ jq -nc '[0, false, [], {}, null, "hello"] | map(type)'
["number","boolean","array","object","null","string"]

select

Filter items by condition.

jq -n '[1,2,3] | map(select(. > 1))'

Conversions

tonumber / toboolean / tostring

Convert between string and number.

jq -n '"42" | tonumber'
jq -n '"true" | toboolean'
jq -n '42 | tostring'

fromjson / tojson

Parse a JSON string or serialize to JSON string.

jq -n '"{\"a\":1}" | fromjson'      # {"a":1}
jq -n '{a:1} | tojson'              # "{\"a\":1}"

Stream generators

empty

Return nothing (zero outputs). Useful as a no-op or to suppress output in conditional branches.

jq -n 'empty'                           # (no output)
jq -n '[1, empty, 2]'                   # [1, 2]
jq -n 'if true then 42 else empty end'  # 42

range

Generate a sequence of numbers.

jq -n 'range(3)'                    # 0 1 2
jq -n 'range(1; 5)'                 # 1 2 3 4
jq -n 'range(0; 10; 3)'             # 0 3 6 9

recurse

Recursively descend into nested structures.

.. is alias for recurse.

jq -n '{a:{b:1}} | recurse'
jq -n '10 | recurse(. - 1; . > 0)'

Stream consumers

add

Sum array elements.

jq -n '[1,2,3] | add'                # 6
jq -n '[] | add'                     # null
jq -n '[{a:1},{a:2}] | add(.[].a)'   # 3

first / last

Return first or last element.

jq -n '[1,2,3] | first'             # 1
jq -n '[1,2,3] | last'              # 3

limit

Limit the number of outputs from a stream.

jq -n '[1,2,3,4,5] | limit(3; .[])'    # 1, 2, 3

reduce

Accumulate array elements into a single value. Takes (init; update) where . is the accumulator and $item is each element.

jq -n '[1,2,3] | reduce .[] as $item (0; . + $item)'      # 6

foreach

Iterate with state and emit a value per element. Takes (init; update; extract).

$ jq -nc '[1,2,3] | foreach .[] as $item (0; . + $item; {idx: ., sum: .})'
{"idx":1,"sum":1}
{"idx":3,"sum":3}
{"idx":6,"sum":6}

any / all

Check if any/all elements satisfy a condition.

jq -n '[1,2,3] | any(. > 2)'       # true
jq -n '[1,2,3] | all(. > 0)'       # true
jq -n '[1,2,3] | any'              # true (any truthy)
jq -n '[] | any'                   # false

Updates

map / map_values

Apply a filter to each array element / object value.

jq -n '[1,2,3] | map(. * 2)'
jq -n '{a:1,b:2} | map_values(. + 1)'

walk

Recursively process all elements in a tree structure.

$ jq -nc '{a:{b:"c"},d:"e"} | walk(if type == "string" then ascii_upcase else . end)'
{"a":{"b":"C"},"d":"E"}

del

Delete a key from an object.

jq -n '{a:1,b:2} | del(.b)'

Paths

pick

Select only specified keys from an object.

jq -n '{a:1,b:2,c:3} | pick(.a, .c)'    # {"a":1,"c":3}

paths / setpath / getpath / delpaths

Inspect and manipulate nested paths.

jq -n '{a:{b:1},c:2} | paths'
jq -n '{a:{b:1},c:2} | paths(scalars)'
jq -n '{} | setpath(["a","b"]; 1)'                  # {"a":{"b":1}}
jq -n '{"a":{"b":1}} | getpath(["a","b"])'          # 1
jq -n '{"a":{"b":1,"c":2}} | delpaths([["a","b"]])' # {"a":{"c":2}}

Membership

contains / inside

Check if a value contains another (strings, arrays, objects).

jq -n '"foobar" | contains("foo")'      # true
jq -n '[1,3,2] | contains([1,2])'       # true
jq -n '{"a":1,"b":2} | contains({a:1})' # true

inside is the inverse — checks if input is inside argument.

jq -n '"foo" | inside("foobar")'    # true

indices

Find all indices where a value/pattern occurs.

jq -n '"abcabc" | indices("b")'     # [1,4]
jq -n '[1,2,1,2] | indices(1)'      # [0,2]

index

Find the first index of a value in an array or string.

jq -n '"abcabc" | index("b")'       # 1
jq -n '[1,2,1,2] | index(1)'        # 0

Arrays

sort / sort_by

Sort array; sort_by(f) sorts by the result of applying f to each element.

jq -n '[3,1,2] | sort'
jq -n '[{"age":2},{"age":1}] | sort_by(.age)'

group_by

Group elements into array of arrays based on a key expression.

$ jq -nc '[{"name":"a","type":"x"},{"name":"b","type":"y"},{"name":"c","type":"x"}] | group_by(.type)'
[[{"name":"a","type":"x"},{"name":"c","type":"x"}],[{"name":"b","type":"y"}]]

group_by(.x) requires prior sort_by(.x)

unique / unique_by

Remove duplicate elements. unique uses value equality; unique_by(f) uses the result of f as the key.

jq -n '[1,2,1,3] | unique'                          # [1,2,3]
jq -n '[{"x":1},{"x":2},{"x":1}] | unique_by(.x)'   # [{"x":1},{"x":2}]

min / max / min_by / max_by

Find minimum or maximum values.

jq -n '[3,1,2] | min'              # 1
jq -n '[3,1,2] | max'              # 3
jq -n '[{"x":2},{"x":1}] | min_by(.x)'  # {"x":1}

reverse

Reverse an array.

jq -n '[1,2,3] | reverse'           # [3,2,1]

transpose

Transpose a matrix (array of arrays).

jq -n '[[1,2],[3,4]] | transpose'   # [[1,3],[2,4]]

flatten

Flatten nested arrays to a specified depth (default: infinite).

jq -n '[[1,[2]],[3]] | flatten'    # [1,2,3]
jq -n '[[1,[2]],[3]] | flatten(1)' # [1,[2],3]

Text strings

ascii_downcase / ascii_upcase

Convert string to lowercase/uppercase.

jq -n '"Hello" | ascii_downcase'    # "hello"
jq -n '"Hello" | ascii_upcase'      # "HELLO"

join

Join array elements with a separator string.

jq -n '[1,2,3] | join(", ")'

Text string formatting

• @json: Format value as JSON string, e.g. jq -nc '42 | @json' → "42"
• @csv: Convert array to CSV line, e.g. jq -nr '["a","b"] | @csv' → "a","b"
• @tsv: Convert array to TSV line, e.g. jq -nr '["a","b"] | @tsv' → a\tb
• @html: Escape HTML special chars, e.g. jq -nr '"<script>" | @html' → <script>
• @uri: Percent-encode URI component, e.g. jq -nr '"hello world" | @uri' → hello%20world
• @base64: Base64 encode string, e.g. jq -nr '"hello" | @base64' → aGVsbG8=
• @base64d: Base64 decode string, e.g. jq -nr '"aGVsbG8=" | @base64d' → hello
• @sh: Escape for shell use (single-quote wrap), e.g. jq -nr '"it'\''s" | @sh' → 'it'\''s'
• @text: Convert to string, e.g. jq -nc '42 | @text' → "42"

Regular expressions

test

Check if string matches regex — returns true/false.

jq -n '"hello 42" | test("\\d+")'
jq -n '"Hello" | test("^[a-z]+$"; "i")'

match

Return match objects with details (offset, length, captures).

$ jq -nc '"abc123" | match("[a-z]+|\\d+"; "g")'
{"offset":0,"length":3,"string":"abc","captures":[]}
{"offset":3,"length":3,"string":"123","captures":[]}

capture

Extract named groups into an object.

$ jq -nc '"2024-01-15" | capture("(?<y>\\d+)-(?<m>\\d+)-(?<d>\\d+)")'
{"y":"2024","m":"01","d":"15"}

split / splits

Split string by regex. split returns array; splits streams results.

$ jq -rn '"a,b" | split(",")'
[
  "a",
  "b"
]

$ jq -rn '"a,b" | splits(",")'
a
b

sub / gsub

Replace matched substrings. sub replaces first match, gsub replaces all.

jq -rn '"hello world" | sub("\\w+"; "***")'     # *** world
jq -rn '"hello world" | gsub("\\w+"; "***")'    # *** ***

Capture groups in replacement via $1, $2, etc.:

jq -rn '"2024-01-15" | sub("(?<y>\\d+)-(?<m>\\d+)-(?<d>\\d+)"; "\(.y).\(.m).\(.d)")'

Date & Time

now

Return the current time as Unix epoch seconds (float).

jq -n 'now'               # 1705314600.22693

fromdate

Parse an ISO 8601 datetime string to Unix epoch seconds.

jq -n '"2024-01-15T10:30:00Z" | fromdate'   # 1705314600

todate

Convert Unix epoch seconds to an ISO 8601 datetime string.

jq -n 'now | todate'                        # "2024-05-18T12:34:56Z"

strftime

Formats a datetime (epoch seconds) to a string using a format specifier.

jq -n 'now | strftime("%Y-%m-%d")'                # "2024-05-18"

strflocaltime

Like strftime but formats in local timezone.

jq -n 'now | strflocaltime("%Y-%m-%d %H:%M:%S")'

strptime

Parses a string to array representation using a format specifier.

jq -n '"2024-01-15" | strptime("%Y-%m-%d")'       # [2024,0,15,0,0,0,1,14]

mktime / gmtime

Convert between array representation and epoch seconds.

jq -n '[2024,0,15,0,0,0,1,14] | mktime'    # 1705305600
jq -n '1705305600 | gmtime'                # [2024,0,15,0,0,0,1,14]

I/O

inputs

Read remaining input values (useful with -n for streaming).

printf '1\n2\n3\n' | jq -n '[inputs] | add'    # 6

debug

Print value to stderr, pass through unchanged. Useful for tracing in pipelines.

jq -n '1, 2 | debug'

stderr output: ["DEBUG:",1] ["DEBUG:",2]

try / catch / error

Handle errors without halting the pipeline. try expr silently suppresses errors; try expr catch handler runs handler with the error message as ..

jq -n '3 | try .a'                              # (nothing)
jq -n '3 | try .a catch "na"'                   # "na"
jq -n 'try error("fail") catch "caught: \(.)"'  # "caught: fail"

$ENV

$ENV is an object representing the environment variables as set when the jq program started. $ENV is readonly

$ENV               Object of all environment variables
$ENV | keys        List all env var names
$ENV.PATH          Access env var via variable

$ARGS.named

Access named arguments passed via --arg / --argjson programmatically.

jq -n --arg a 1 --argjson b 2 '$ARGS.named'    # {"a":"1","b":2}

$ARGS.positional

Access positional arguments passed via --args / --jsonargs programmatically.

jq -n '$ARGS.positional' --args hello world  # ["hello","world"]
jq -n '$ARGS.positional' --jsonargs 1 2 3    # [1,2,3]

Defining Functions

def add1: . + 1;
def map(f): [.[] | f];          # f is a filter parameter (def filter)
def add(x; y): x + y;           # Multi-arg (note semicolons)