Introduction

My personal notes. The notes divided into following blocks:

• Collections: my favorite command-line tools and utility websites.
• Languages: my learning notes on some programming languages.
• Tools: my experience with famous tools.

Collections

A collections of tools, services, websites.

• CLI Tools
• SASS Tools
• Utility Websites

Cli Tools

dufs

A file server that supports static serving, uploading, searching, accessing control, webdav...

croc

Easily and securely send things from one computer to another with cli

frp

A proxy tools supports intranet penetration, port forwarding.

rg

A better grep

webhookd

Use shell scripts to handle webhook request.

proxychains-ng

Run command with proxy network

proxychains-windows

Run command with proxy network in windows pc.

mdbook

Create book from markdown files.

argc

Easily create and use cli that based on bashscript.

projclean

Find and clean dependencies & builds from software projects to saving space.

rclone

Rsync for cloud storage

bat

Cat with syntax highlight

jless

A command-line JSON viewer

git-cliff

Create changelog from git commit.

plantuml

Code as diagrams

zellij

A terminal multiplexer similar to tmux.

mcfly

Search shell history

zellij

Terminal multiplexer

yq

Process YAML, JSON, XML, CSV and properties documents from the CLI

Sass Tool

gogs

A painless self-hosted Git service

minio

A self-hosted cloud storage compatible with aws s3

meilisearch

A self-hosted search engine.

gorse

A self-hosted recommand system.

dtm

A self-hosted distributed transaction service supports saga, tcc, xa, 2-phase message, outbox patterns.

centrifugo

A self-hosted scalable real-time messaging server.

dynimgen

A self-hosted dynamic image generator.

Utility Websites

amp-what

Discover Unicode Character Entities & Symbols.

webhook.site

Test, process and transform HTTP requests.

draw.io

Draw wireframes, support local storage, cloud storage.

asciiflow

Draw ascii diagrams.

iconfont.cn

Pick icons to create your own icon fonts.

getimg

Fast and Free Online Image Processing Tool.

Languages

Some notes on programing languages.

• Shell
• Rust
• Markdown
• Nodejs

Shell

Variables

# Default value
echo ${value:-xa} # xa
value=xyz/ijk/abc
# Head
echo ${value%/*} # xyz/ijk
echo ${value%%/*} # xyz
# Tail
echo ${value#*/} # ijk/abc
echo ${value##*/} # abc
# Truncate
echo ${value:7} # /abc
echo ${value:7:4} # /abc
echo ${value:(-4):4} # /abc
# Replace
echo ${value/abc/cba} # xyz/cba

Shell Variables

• $#: count args
• $n: nth arg
• $?: last exit code
• $!: last pid

Array

Fruits=('Apple' 'Banana' 'Orange' 'Pear')

echo ${Fruits[0]}           # Element #0
echo ${Fruits[-1]}          # Last element
echo ${Fruits[@]}           # All elements, space-separated
echo ${#Fruits[@]}          # Number of elements
echo ${#Fruits}             # String length of the 1st element
echo ${#Fruits[3]}          # String length of the Nth element
echo ${Fruits[@]:3:2}       # Range (from position 3, length 2)
echo ${!Fruits[@]}          # Keys of all elements, space-separated



Fruits=("${Fruits[@]}" "Grape")         # Push
Fruits+=('Watermelon')                  # Also Push
Fruits=( ${Fruits[@]/Ap*/} )            # Remove by regex match
unset Fruits[2]                         # Remove one item
Fruits=("${Fruits[@]}")                 # Duplicate
Fruits=("${Fruits[@]}" "${Veggies[@]}") # Concatenate

TempFiles=($(ls /tmp))

for i in "${Fruits[@]}"; do
  echo $i
done

if [[ " ${Fruits[*]} " =~ " Banana " ]]; then
  echo "Apple in Fruits"
fi

Dictionary

declare -A sounds

sounds[dog]="bark"
sounds[cow]="moo"
sounds[bird]="tweet"
sounds[wolf]="howl"

echo ${sounds[dog]} # Dog's sound
echo ${sounds[@]}   # All values
echo ${!sounds[@]}  # All keys
echo ${#sounds[@]}  # Number of elements
unset sounds[dog]   # Delete dog

# Value loop
for val in "${sounds[@]}"; do
  echo $val
done

# Key loop
for key in "${!sounds[@]}"; do
  echo $key
done

Condition

test

if

if [[ "foo" == "$1" ]]; then
  echo foo
elif [[ "bar" == "$1" ]]; then
  echo bar
else
  echo "not found"
  exit 1
fi

case

case "$1" in
  start | up)
    vagrant up
    ;;
  stop | down)
    vagrant up
    ;;

  *)
    echo "Usage: $0 {start|stop}"
    ;;
esac

exitcode

if grep -q 'alice' /etc/passwd; then
  echo "User alice exists"
fi

Loop

# index
for i in `seq 1 10`; do
  echo $i 
done

for i in `seq 1 2 10`; do
  echo $i
done

# fs
for f in ./*; do
  echo $f
done

# array
arr=( a b c )
for v in ${arr[@]}; do
  echo $v
done

# lines
cat file.txt | while read l; do
  echo $l
done

# forever
while true; do
  sleep 1
done

Function

Returns value

foo() {
    local msg='world'
    echo "$1 $msg"
}
result="$(foo hello)"
echo $result

Returns boolean

_is_win() {
    if [[ "$(uname)" =~ "_NT" ]]; then
        return 0
    else
        return 1
    fi
}

if _is_win; then
  echo 'windows'
fi

Value

Condition

Math

a=$(($RANDOM%200))  # Random number 0..199
$((a + 200))        # Add 200 to $a

Redirect

• >: equal to 1>
• 1>&2： redirect stdout to stderr
• 2>&1： redirect stderr to stdout
• >& file: redirect stdout/stderr to file

Heredoc

cat << EOF | sudo tee -a /etc/sudoers.d/$USER
$USER ALL=(ALL) NOPASSWD:ALL
EOF 

• <<EOF: will replace variable
• <<'EOF': not replace variable
• <<-EOF: will replace variable trim \t
• <<-'EOF': not replace variable，trim \t

Yes/No

read -p "Are you sure (y/n)? " choice
if [ "$choice" = "y" ]; then
  echo choice yes
fi

Common Tools

• Character related: find/sed/awk
• Network related: curl/wget
• Argument parser related: getopt/argc
• JSON/YAML/TOML related: jq/yq
• Interative related: gum

Rust

Installation

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Rustup

Concepts

• target: x86_64-pc-windows-msvc, x86_64-unknown-linux-gnu, wasm32-unknown-unknown
• toolchain: stable-x86_64-pc-windows-msvc, nightly-2021-05-11-x86_64-unknown-linux-gnu
• component: rustc,rustsrc,rustfmt, rust-analyzer

Snippets

rustup update 

rustup toolchain install nightly

rustup target add wasm32-unknown-unknown --toolchain nightly

rustup default nightly

rustup override set nightly

Crates

• tokio: Async
• crossbeam: Concurrency
• rayon: Concurrency computation
• axum: web framework
• poem: web framework，supports openapi
• sqlx: sql
• sea-orm: orm
• reqwest: http client
• anyhow: generic error framework
• thiserror: library error framework
• clap: cli framework
• nom: bytes/charactor parser
• assert_cmd: cli test framework
• insta: snapshot test framework

Cargo

Plugins

• cargo-outdated: Displaying when dependencies have newer versions available.
• cargo-audit: Audit your dependencies for crates with security vulnerabilities.
• cargo-udeps: Find unused dependencies in Cargo.toml
• cargo-bloat: Find out what takes most of the space in your executable.
• cargo-fuzz: Fuzzing with libFuzzer
• cargo-insta: Snapshot testing
• cargo-binstall: Binary installation for rust projects
• cargo-geiger: Detects usage of unsafe Rust in a Rust crate and its dependencies.

Snippets

cargo add serde
cargo add serde -F derive
cargo add -D serde
cargo add -B serde

cargo tree
cargo tree -p libc
cargo tree -p libc -i

Cargo.toml

[profile.release]
lto = true
strip = true
opt-level = "z"

Resources

• Rust Language Cheat Sheet
• Rust by Example
• Rust Design Patterns
• Rust Atomics and Locks
• learn project: toydb
• learn project: mini-redis
• Rust API Guidelines
• pretzelhammer Blog
• Jon Gjengset Youtube

Markdown

Heading

Heading3

Heading4

Heading5

Text Style

bold text

italicized text

blockquote

Ordered List

1. First item
2. Second item
3. Third item

Unordered List

• First item
• Second item
• Third item

Code

Inline code with backticks

#![allow(unused)]
fn main() {
#[derive(Debug)]
pub enum State {
    Start,
    Transient,
    Closed,
}

impl From<&'a str> for State {
    fn from(s: &'a str) -> Self {
        match s {
            "start" => State::Start,
            "closed" => State::Closed,
            _ => unreachable!(),
        }
    }
}
}

Link

Link

Image

Horizontal Rule

Autolink literals

www.example.com, https://example.com, and contact@example.com.

Footnote

A note¹

Strikethrough

one or two tildes.

Table

Tasklist

• to do
• done

Math

$E = mc^2 + 2$

Nodejs

Installation

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash

nvm i --lts

nvm i --lts --reinstall-packages-from=node

NPM

# list all packages
npm ls -g --depth=0
# init projects
npm i -y

Run node service

1. Use pm2

pm2 start --name app npm start

2. Use systemd

[Unit]
After=network.target

[Service]
Environment=NODE_PORT=3000
Type=simple
User=ubuntu
ExecStart=/usr/bin/node /home/ubuntu/app.js
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo cp app.service /etc/systemd/system/
sudo systemd daemon-reload
sudo systemd start app

3. Use docker

FROM sigoden/node:native as builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --prod

FROM sigoden/node:slim
WORKDIR /app
COPY --from=builder /app .
COPY . .
EXPOSE 3000
CMD ["node", "index.js"]

Native addon

1. INstall deps like gcc, make, python

# ubuntu
apt install build-essential

# alpine
apk add make gcc g++ python3 git

2. Test native addon

npm i --build-from-source bcrypt

ES features

https://node.green

Tools

Some notes on tools.

• Docker
• Git
• Github CI
• Linux
• Redis

Docker

Installation

curl -fsSL get.docker.com | sudo bash

None-Root

sudo usermod -aG docker $USER

Snippets

# Run container then enter shell
docker run -it --rm -v `pwd`/data:/data -p 3000:3000 node:16 bash

# Run container as service
docker run -d --name redis -p 6379:6379 redis:5

# Exec in container
docker exec -it redis bash
docker exec redis bash -c 'redis-cli set k1 a && redis-cli get k1'

# View log
docker logs --tail 50 -f redis

# Clean
docker system prune -f

# Clean dead images
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

Use Mirror

• WIN: Docker Desktop -> Preferences -> Docker Engine
• LINUX: /etc/docker/daemon.json

{
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn"]
}

Dockerfile

COPY vs ADD

• ADD copy files, download remote resources, handle zip/tar.gz
• COPY only copy local files

Do not use ADD unless COPY not works.

ADD /source/file/path  /destination/path
ADD http://source.file/url  /destination/path
ADD source.file.tar.gz /destination

COPY /source/file/path  /destination/path

ENTRYPOINT vs CMD

• No ENTRYPOINT

• ENTRYPOINT e1

• ENTRYPOINT ["e1"]

Conconcate commands

RUN set -ex \
    # comment
    && apt-get update && apt-get install -y curl --no-install-recommends \
    && rm -rf /var/lib/apt/lists/* 

Layers

FROM sigoden/node:native as builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --prod

FROM sigoden/node:slim
WORKDIR /app
COPY --from=builder /app .
COPY . .
EXPOSE 3000
CMD ["node", "index.js"]

Heredoc

RUN printf '#!/bin/bash\n\
echo 1\n\
echo 2'\
>> /tmp/hello

Enrypoint file

#!/bin/sh
set -e

if [ "${1#-}" != "${1}" ] || [ -z "$(command -v "${1}")" ]; then
  set -- app "$@" # change app to your binary
fi

exec "$@"

Normal User

RUN groupadd --gid 1000 user \
  && useradd --uid 1000 --gid user --shell /bin/bash --create-home user

Multi-Arch Build

docker buildx create --name mybuilder --driver docker-container
docker buildx use mybuilder

docker buildx build \
  --platform linux/amd64,linux/arm64,linux/arm/v7,linux/386 \
  -t demo . 

Git

Stages

Clone

# branch
git clone -b dev https://github.com/org/repo.git
# target folder
git clone https://github.com/org/repo.git myrepo 
# contains submodule
git clone --recurse-submodules https://github.com/org/repo.git
# abaddon history
git clone --depth=1 https://github.com/org/repo.git

Branch

# list branches
git branch
# list remote branches
git branch -r
# list all branches
git branch -a
# create branch
git checkout -b feat1
# rename a branch
git branch -m newname
# checkout branch
git checkout feat1
# checkout remote branch
git checkout -t origin/dev
# delete branch
git branch -d feat1
# delete branch forcedly
git branch -D feat1
# delete branch of remote repo
git push origin :feat1

Tag

# list tags
git tag
# create tag
git tag v1.0.0
# delete tag
git tag -d v1.0.0
# delete tag forcedly
git tag -D v1.0.0
# delete tag of remote repo
git push origin :v1.0.0

Fetch

# pull changes
git fetch
# pull github pr
git fetch origin pull/ID/head:BRANCH_NAME
# pull changes and prune none-exist remote branches
git fetch --purge

Snippets

# merge last commit
git commit --amend
# sync submodule
git submodule update --init --recursive
# inspect remote url
git remote -v
# change remote url
git remote set-url origin $new_repo

Config

User

git config --global user.name $user
git config --global user.email $email

Ignore

# linux/macos
git config --global core.excludesFile '~/.gitignore'
# windows
git config --global core.excludesFile "$env:USERPROFILE\.gitignore"

Cjk path

git config --global core.quotepath false

CRLF

git config --global core.eol lf
git config --global core.autocrlf input

Credentials

WSL2

git config --global credential.helper \
  "/mnt/c/Program\\ Files/Git/mingw64/bin/git-credential-manager.exe"

SSH Repo

sudo useradd -m -r -U -d /home/git-repos git
sudo su - git

mkdir -p ~/.ssh
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
touch ~/.ssh/authorized_keys

git init --bare ~/repo.git

git remote add origin git@<your-server-ip>:repo.git

Github CI

Triggers

on: 
  push:
    tags: 
      - v[0-9]+.[0-9]+.[0-9]+*

on
  push:
    branches:
    - main
  pull_request:

on:
  workflow_dispatch:
  schedule:
    - cron:  '0 2 * * *'

Environment Variables

jobs:
  job1:
    runs-on: ubuntu-latest
    env:
      FOO: foo
    steps:
      - if: ${{ env.FOO == 'foo' }}
        run: echo "$BAR"
        env:
          BAR: bar

Debug Tips

Print all environment variables

    steps:
      - run: env

Print github variables

    steps:
      - run: echo "$GITHUB_CONTEXT"
        env:
          GITHUB_CONTEXT: ${{ toJSON(github) }}

Print shell evalutaion

    steps:
      - run: |
          set -x 

          src=`pwd`
          dist=$src/dist
        shell: bash

ssh into runner

    steps:
    - name: Setup upterm session
      uses: lhotari/action-upterm@v1

RUNNER will log:

=== S1WV8GFLUQTD7SN7TAKM                                                                 
Command:                tmux new -s upterm -x 132 -y 43                                 
Force Command:          tmux attach -t upterm                                           
Host:                   ssh://uptermd.upterm.dev:22                                     
SSH Session:            ssh s1wv8gfLuqTd7sn7taKm:MTAuMjQ0LjAuNzY6MjI=@uptermd.upterm.dev

Get Tag info

jobs:
  release:
    outputs:
      rc: ${{ steps.check-tag.outputs.rc }}
    steps:
      - name: Get Tag
        id: get_tag
        shell: bash
        run: |
          tag=${GITHUB_REF##*/}
          if [[ "$tag" =~ [0-9]+.[0-9]+.[0-9]+$ ]]; then
            echo "rc=false" >> $GITHUB_OUTPUT
          else
            echo "rc=true" >> $GITHUB_OUTPUT
          fi

Add environment variable

jobs:
  job1:
    steps:
      - name: Set env var
        run: |
          echo "foo=bar" >> $GITHUB_ENV

Pass variables between jobs

jobs:
  job1:
    runs-on: ubuntu-20.04
    outputs:
      tag: ${{ steps.get_tag.outputs.tag }}

  job2:
    needs: ["job1"]
    if: needs.job1.outputs.tag != 'null'

Pass artificats between jobs

jobs:
  job1:
    steps:
      - uses: actions/upload-artifact@v4
        with:
          name: app
          path: target/release/app

  job2:
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: app
          path: ${{ github.workspace }}/target/release/app 

Build and push docker image

jobs:
  docker:
    name: Publish to Docker Hub
    if: startsWith(github.ref, 'refs/tags/')
    runs-on: ubuntu-latest
    needs: release
    steps:
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          build-args: |
            REPO=${{ github.repository }}
            VER=${{ github.ref_name }}
          platforms: |
            linux/amd64
            linux/arm64
            linux/386
            linux/arm/v7
          push: ${{ needs.release.outputs.rc == 'false' }}
          tags: ${{ github.repository }}:latest, ${{ github.repository }}:${{ github.ref_name }}

Linux

Inspect

File

# List files
ls -alh file

# Modify user group
chown -R alice file
chown -R alice:www-data file

# Modify perms
chmod 600 file
chmod -w file
chmod +x file
chmod o-x file
chmod u+w file

# Open with default app
xdg-open file

# link file
ln -s /user/bin/vim /usr/bin/vi
# Link file relative
ln -sr node_modules/eslint/bin/eslint.js node_modules/.bin/eslint

# Zip
zip -r data.zip data
# Inspect zip files
unzip -t data.zip
# Unzip file
unzip data.zip
# Unzip to specific folder
unzip -d output-dir data.zip
# Unzip with pass
zip -er data.zip data

# Tar
tar -cf data.tar.gz data
# Inspect tar files
tar -tf data.tar.gz
# Untar
tar -xf data.tar.gz
# Untar to specific folders
tar -xf data.tar.gz -C output-dir

# bz format compression (slower but smaller)
tar -cjf data.tar.bz2 data

# Download
wget $url
wget -O /tmp/file $curl
curl -o /tmp/file $curl

# Find
find -name README.md
find / -name sfz
find -type -f -name '*.log' -exec rm -rf {} \;

# split by number of copies
split -d -n 5 data.zip data.zip.part
# split by size
split -d -b 256M data.zip data.zip.part
# Combine split files
cat data.zip.part* > data.zip

# generate file
dd if=/dev/zero of=/tmp/1G.bin bs=1M count=1000
dd if=/dev/urandom of=/tmp/1G.bin bs=10M count=1000

User/Group

# add users (more complete, similar to interface operations)
adduser alice
# add user (recommended version)
useradd -m alice
# add user, specify user directory
useradd -d /d/alice alice
# add user, specify uid, gid
useradd -u 1001 -g 1001 nodejs
# add program account
useradd -r -s /usr/sbin/nologin nodejs

# MODIFY THE USER SHELL
usermod -s /usr/bin/bash alice

# user join group
usermod -a -G docker alice

# modify user password
passwd alice

# delete users
userdel nodejs

# add group
groupadd -g 344 linuxde
# delete group
groupdel linuxde

Process

# find process
ps aux | grep nginx

# kill process
kill $pid
# kill process forcelly
kill -9 $pid
# kill process by name
pkill nginx
pkill '^ssh$'
pkill -9 nginx

# find process by port
lsof -i:3000
netstat -utlpn | grep 3000
# find port by process
netstat -utlpn | grep nginx

cat /proc/$pid/status | grep VmRSS
cat /proc/$pid/cmdline 
cat /proc/$pid/environ 

ls -l /proc/$pid/cwd
ls -l /proc/$pid/exe
ls -l /proc/$pid/root

# run in backgroud
nohup server &
nohup server > server.log 2>&1 &
nohup server > server.log 2> server.err &

Text

# count lines
cat file | wc -l

# first n lines
cat file | head -3
# last n lists
cat file | tail -3

cat file | more

# hex
cat file | xxd 
cat file | xxd -p > file.hex

# split columns with `:`, only show first, third columns
cat /etc/group | cut -d: -f1,3
# only show 5-10 charactors of each row
cat file | cut -c5-10
# only show first 10 charactors of each row
cat file | cut -c-10
# only show 10th and subsequent of each line
cat file | cut -c10-

# replace charactor
cat file | tr '\t' ' '
# delete charactor
cat file | tr -d '\r'

# remove duplicates
cat file | sort | uniq 
# count duplicates
cat file | sort | uniq -c
# show duplicates
cat file | sort | uniq -d

sed

# delete first line
cat file | sed '1d'
# delete empty line
cat file | sed '/^$/d'
# insert at first line
cat file | sed '1i#!/usr/bin/env node\n'
# append on last line
cat file | sed '$a\\n'
# replace functions
cat file | sed '/foo() {/,/}/c foo() { }'
# replace path
cat file| sed 's|/bin/sh|/bin/bash|g'
# replace with matched parts
cat file | sed 's|\(\w\+\) \(\w\+\) \(\w\+\)|\2 \3 \1|'
# change file
sed -i 's/\w\+.ubuntu.com/mirrors.tuna.tsinghua.edu.cn/' /etc/apt/sources.list

awk

# print specific column
docker images | awk '{print $1}'
# specific delimeter
cat /etc/paswd | awk -F: '{print $1}'
# combine columns
cat file | awk '{print $1,$3}'

Regex

GLOB

Terminal

• !! last command
• !$ last postional parameter of last command

Set

set -euxo pipefail

• -e: Exit on none-zero status
• -x: Print a trace
• -u: Treat unset variables as error
• -o pipefail: Return none-zero if last pipeline return none-zero

Xargs

Capture the output of one command and pass it to another command

# Batch renames
ls -1 . | sed 's/\.mdx$//' | xargs -I{} mv {}.mdx {}.md

Cron Syntax

Min  Hour Day  Mon  Weekday
*    *    *    *    *  command to be executed
┬    ┬    ┬    ┬    ┬
│    │    │    │    └─  Weekday  (0=Sun .. 6=Sat)
│    │    │    └──────  Month    (1..12)
│    │    └───────────  Day      (1..31)
│    └────────────────  Hour     (0..23)
└─────────────────────  Minute   (0..59)

Common Snippets

Script Directory

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"

Sudo Without Pass

cat << EOF | sudo tee -a /etc/sudoers.d/$USER
$USER ALL=(ALL) NOPASSWD:ALL
EOF 

Ulimit

cat <<EOF | sudo tee -a /etc/security/limits.d/$USER.conf
$USER soft nproc 100000
$USER hard nproc 100000
$USER soft nofile 100000
$USER hard nofile 100000
EOF

Change repo source

ubuntu

sed -i -E 's/\w+.ubuntu.com/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list

Alpine

sed -i 's/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apk/repositories

Load environment variables from .env

eval $(cat .env | sed 's/^/export /')

Redis

Clear all

EVAL "return redis.call('del', unpack(redis.call('keys', ARGV[1])))" 0 prefix:*

Enter redis container

#!/bin/bash -e
server=$(docker ps | grep redis | awk  '{print $1}')
docker exec -it $server redis-cli -a ${1:-pass}