Shell Scrap
Table of Contents
- Flow Control
- Keyboard Input
- Parameters
- Errors and Signals and Traps
- Creating safe temporary files
- Debug
- Parameter expansion
- String Operators
- Command
- getopts
- compute float number
- positional parameters
- 多个文件总大小
- diff ignore files only in one directory
- regular expression
- touch
- the path of directory in which a Bash script is located
- only one instance of a script is running at a time (mutual exclusion, locking
- cc
Flow Control
if
- syntax
# First form if condition ; then commands fi # Second form if condition ; then commands else commands fi # Third form if condition ; then commands elif condition ; then commands fi
- Example
if [ -f .bash_profile ]; then echo "You have a .bash_profile. Things are fine." else echo "Yikes! You have no .bash_profile!" fi
- Condition
Expression Description
- -d file True if file is a directory.
- -e file True if file exists.
- -f file True if file exists and is a regular file.
- -L file True if file is a symbolic link.
- -r file True if file is a file readable by you.
- -w file True if file is a file writable by you.
- -x file True if file is a file executable by you.
- file1 -nt file2 True if file1 is newer than (according to modification time) file2
- file1 -ot file2 True if file1 is older than file2
- -z string True if string is empty.
- -n string True if string is not empty.
- string1 = string2 True if string1 equals string2.
- string1 != string2 True if string1 does not equal string2.
- -a (AND) and -o (OR). The -a and -o operators are similar to the && and || operators used with exit statuses.
case
- syntax
case word in patterns ) statements ;; esac
- example
#!/bin/bash echo -n "Enter a number between 1 and 3 inclusive > " read character case $character in 1 ) echo "You entered one." ;; 2 ) echo "You entered two." ;; 3 ) echo "You entered three." ;; * ) echo "You did not enter a number" echo "between 1 and 3." esac
Keyboard Input
read
- example
#!/bin/bash echo -n "Enter some text > " read text echo "You entered: $text"
- option
-t
(time)The -t option followed by a number of seconds provides an automatic timeout for the read command.
#!/bin/bash echo -n "Hurry up and type something! > " if read -t 3 response; then echo "Great, you made it in time!" else echo "Sorry, you are too slow!" fi
- option
-s
The -s option causes the user's typing not to be displayed.
Parameters
$#
the shell maintains a variable called $# that contains the number of items on the command line
shift
shift
is a shell builtin that operates on the positional parameters.
Each time you invoke shift, it "shifts" all the positional parameters
down by one. $2 becomes $1, $3 becomes $2, $4 becomes $3, and so on.
Errors and Signals and Traps
Creating safe temporary files
The preferred technique is to write them in a local directory such as
~/tmp
(a tmp subdirectory in the user's home directory.) If you must
write files in /tmp
, you must take steps to make sure the file names
are not predictable. Predictable file names allow an attacker to
create symbolic links to other files that the attacker wants you to
overwrite.
example
TEMP_FILE=$TEMP_DIR/printfile.$$.$RANDOM
$$
shell variable to embed the process id (pid) of the program$RANDOM
shell variable to append a random number to the file name
Debug
Watching your script run
- Method1
#!/bin/bash -x
- Method2
you can use the set command within your script to turn tracing on and off. Use
set -x
to turn tracing on andset +x
to turn tracing off. For example.:#!/bin/bash number=1 set -x if [ $number = "1" ]; then echo "Number equals 1" else echo "Number does not equal 1" fi set +x
trap command
其基本的语法是:
trap 'command' signal
其中signal是要捕获的信号,command是捕获到指定的信号之后,所要执行的命令。可以用kill –l命令看到系统中全部可用的信号名,捕获信号后所执行的命令可以是任何一条或多条合法的shell语句,也可以是一个函数名。
shell脚本在执行时,会产生三个所谓的“伪信号”,(之所以称之为“伪信号”是因为这三个信号是由shell产生的,而其它的信号是由操作系统产生的),通过使用trap命令捕获这三个“伪信号”并输出相关信息对调试非常有帮助。
EXIT 从一个函数中退出或整个脚本执行完毕 ERR 当一条命令返回非零状态时(代表命令执行不成功) DEBUG 脚本中每一条命令执行之前
tee command
tee命令会从标准输入读取数据,将其内容输出到标准输出设备,同时又可将内容保存成文件。
运行这个脚本,实际输出的却不是本机的ip地址,而是广播地址,这时我们可以借助tee命令,输出某些中间结果,将上述脚本片段修改为:
ipaddr=`/sbin/ifconfig | grep 'inet addr:' | grep -v '127.0.0.1' | tee temp.txt | cut -d : -f3 | awk '{print $1}'` echo $ipaddr
之后,将这段脚本再执行一遍,然后查看temp.txt文件的内容:
$ cat temp.txt inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
use debug hook
使用DEBUG宏来控制是否要输出调试信息,在shell脚本中我们同样可以使用这样的机制,如下列代码所示:
if [ “$DEBUG” = “true” ]; then echo “debugging” #此处可以输出调试信息 fi
DEBUG() { if [ "$DEBUG" = "true" ]; then $@ fi } a=1 DEBUG echo "a=$a"
command
-n 只读取shell脚本,但不实际执行
-x 进入跟踪方式,显示所执行的每一条命令
-c "string" 从strings中读取命令 {sh -c 'a=1;b=2;let c=$a+$b;echo "c=$c"'}
- 对"-x"选项的增强
几个shell内置的环境变量:
- $LINENO 代表shell脚本的当前行号,类似于C语言中的内置宏_LINE__
- $FUNCNAME 函数的名字,类似于C语言中的内置宏_func__,但宏_func_只能代表当前所 在的函数名,而$FUNCNAME的功能更强大,它是一个数组变量,其中包含了整个调用链上所有的函数的名字,故变量${FUNCNAME1}代表shell脚本当前正在执行的函数的名字,而变量${FUNCNAME2}则代表调用函数${FUNCNAME1} 的函数的名字,余者可以依此类推。
- $PS4 $PS4的值将被显示在“-x”选项输出的每一条命令的前面
$ export PS4='+{$LINENO:${FUNCNAME[0]}} '
Parameter expansion
Simple usage
$PARAMETER ${PARAMETER}
Indirection
${PARAMETER}
The referenced parameter is not PARAMETER itself, but the parameter
whose name is stored as the value of PARAMETER. If the parameter
PARAMETER has the value "TEMP", then ${!PARAMETER}
will expand to the
value of the parameter named TEMP
Case modification
${PARAMETER^} #modifies the first character to uppercase, ${PARAMETER^^} # all characters are converted ${PARAMETER,} # , operator to lowercase ${PARAMETER,,} ${PARAMETER~} #~ reverses the case of first letter of words in the variabl ${PARAMETER~~}
Variable name expansion
${!PREFIX*} #This expands to a list of all set variable names beginning with the string PREFIX. ${!PREFIX@}
Use a default value
If the parameter PARAMETER is unset (never was defined) or null (empty), this one expands to WORD, otherwise it expands to the value of PARAMETER, as if it just was ${PARAMETER}. If you omit the : (colon), like shown in the second form, the default value is only used when the parameter was unset, not when it was empty.
${PARAMETER:-WORD} ${PARAMETER-WORD}
Assign a default value
This one works like the using default values, but the default text you give is not only expanded, but also assigned to the parameter, if it was unset or null. Equivalent to using a default value, when you omit the : (colon), as shown in the second form, the default value will only be assigned when the parameter was unset.
${PARAMETER:=WORD} ${PARAMETER=WORD}
Use an alternate value
This form expands to nothing if the parameter is unset or empty. If it is set, it does not expand to the parameter's value, but to some text you can specify:
${PARAMETER:+WORD} ${PARAMETER+WORD} echo "The Java application was installed and can be started.${JAVAPATH:+ NOTE: JAVAPATH seems to be set}"
Display error if null or unset
If the parameter PARAMETER is set/non-null, this form will simply expand it. Otherwise, the expansion of WORD will be used as appendix for an error message:
- an interactive shell has $? to a non-zero value
- a non-interactive shell exits with a non-zero exit code
${PARAMETER:?WORD} ${PARAMETER?WORD}
Arithmetic and command expansion
#Arithmetic expansion $(( EXPRESSION )) $[ EXPRESSION ] #Please do not use the second form $[ … ]! It's deprecated. #Command substitution $( COMMAND ) ` COMMAND `
Brace expansion
{string1,string2,...,stringN} {<START>..<END>} <PREAMBLE>{........} {........}<POSTSCRIPT> <PREAMBLE>{........}<POSTSCRIPT>
String Operators
String length
${#PARAMETER}
The second is to use builtin function expr
,
expr length $string # or expr "$string" : '.*'
For checking if length of the string is zero you can use -n STRING
operator.
stringZ=abcABC123ABCabc echo ${#stringZ} # 15 echo `expr length $stringZ` # 15 echo `expr "$stringZ" : '.*'` # 15
Determining the Length of Matching Substring at Beginning of String
expr match "$string" '$substring'
$substring
is a regular expression.
my_regex=abcABC123ABCabc echo `expr match "$my_regex" 'abc[A-Z]*.2'` # 8 echo `expr "$my_regex" : 'abc[A-Z]*.2'` # 8
Index
Function index
return the position of substring in string counting
from one and 0 if substring is not found.
expr index $string $substring
stringZ=abcABC123ABCabc echo `expr index "$stringZ" C12` # 6 # C position.
Substr
expr substr $string $position $length #Extracts $length characters from $string starting at $position. stringZ=abcABC123ABCabc # 1-based indexing. echo `expr substr $stringZ 1 2` # ab
substring function is also available as a part of pattern matching operators
${string:position} ${string:position:length}
stringZ=abcABC123ABCabc # 0-based indexing. echo ${stringZ:0} # abcABC123ABCabc echo ${stringZ:1} # bcABC123ABCabc echo ${stringZ:7:3} # 23A
Search and Replace
${var:pos[:len]} # extract substr from pos (0-based) for len ${var/substr/repl} # replace first match ${var//substr/repl} # replace all matches ${var/#substr/repl} # replace if matches at beginning (non-greedy) ${var/##substr/repl} # replace if matches at beginning (greedy) ${var/%substr/repl} # replace if matches at end (non-greedy) ${var/%%substr/repl} # replace if matches at end (greedy) ${#var} # returns length of $var ${!var} # indirect expansion
Concatenation
Strings can be concatenated by juxtaposition and using double quoted strings. For example
PATH="$PATH:/usr/games"
Trimming from left and right
Using the wildcard character (?)
:
test="~/bin/" trimmed_last=${test%?} trimmed_first=${test#?} echo "original='$test,timmed_first='$trimmed_first', trimmed_last='$trimmed_last'" # original='~/bin/,timmed_first='/bin/', trimmed_last='~/bin'
Assignment of default value for undefined variables
Operator: ${var:-bar}
is useful for assigning a variable a default value.
$ export var="" $ echo ${var:-one} one $ echo $var
set variable if it is not defined with the operator ${var:=bar}
$ export var="" $ echo ${var:=one} one echo $var one
Pattern Matching
${var#t*is} # Deletes the shortest possible match from the left export $var="this is a test" echo ${var#t*is} is a test
${var##t*is} # Deletes the longest possible match from the left export $var="this is a test" echo ${var##t*is} a test
${var%t*st} # Deletes the shortest possible match from the right export $var="this is a test" echo ${var%t*st} this is a
${var%%t*st} # Deletes the longest possible match from the right export $var="this is a test" echo ${var%%t*is}
Command
!$
!$是一个特殊的环境变量,它代表了上一个命令的最后一个字符串。如:你可能会这样:$mkdir mydir $mv mydir yourdir $cd yourdir # 可以改成: $mkdir mydir $mv !$ yourdir $cd !$ sudo !! # 以root的身份执行上一条命令 。
- cd –回到上一次的目录 。场景举例:当前目录为/home/a,用cd ../b切换到/home/b。这时可以通过反复执行cd –命令在/home/a和/home/b之间来回方便的切换。
- ‘ALT+.’ or ‘<ESC> .’热建alt+. 或 esc+. 可以把上次命令行的参数给重复出来。 ^oldnew 替换前一条命令里的部分字符串。场景:echo "wanderful",其实是想输出echo "wonderful"。只需要ao就行 了,对很长的命令的错误拼写有很大的帮助。
- du -s * | sort -n | tail 列出当前目录里最大的10个文件。
:w !sudo tee %
在vi中保存一个只有root可以写的文件date -d@1234567890
时间截转时间> file.txt
创建一个空文件,比touch短。mtr coolshell.cn
mtr命令比traceroute要好。在命令行前加空格,该命令不会进入history里。echo “ls -l” | at midnight
在某个时间运行某个命令。curl -u user:pass -d status=”Tweeting from the shell” http://twitter.com/statuses/update.xml
命令行的方式更新twitter。curl -u username –silent “https://mail.google.com/mail/feed/atom” | perl -ne ‘print “\t” if /<name>/; print “$2\n” if /<(title|name)>(.*)<\/\1>/;’
检查你的gmail未读邮件ps aux | sort -nk +4 | tail
列出头十个最耗内存的进程man ascii
显示ascii码表。ctrl-x e
快速启动你的默认编辑器(由变量$EDITOR设置)。netstat –tlnp
列出本机进程监听的端口号。(陈皓注:netstat -anop 可以显示侦听在这个端口号的进程)- tail
tail -f /path/to/file.log | sed '/^Finished: SUCCESS$/ q'
当file.log里出现Finished: SUCCESS时候就退出tail,这个命令用于实时监控并过滤log是否出现了某条记录。
ssh user@server bash < /path/to/local/script.sh
在远程机器上运行一段脚本。这条命令最大的好处就是不用把脚本拷到远程机器上。ssh user@host cat /path/to/remotefile | diff /path/to/localfile -
比较一个远程文件和一个本地文件net rpc shutdown -I ipAddressOfWindowsPC -U username%password
远程关闭一台Windows的机器screen -d -m -S some_name ping my_router
后台运行一段不终止的程序,并可以随时查看它的状态。-d -m参数启动“分离”模式,-S指定了一个session的标识。可以通过-R命令来重新“挂载”一个标识的 session。更多细节请参考screen用法 man screen。wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com
下载整个www.example.com网站。curl ifconfig.me
当你的机器在内网的时候,可以通过这个命令查看外网的IP。convert input.png -gravity NorthWest -background transparent -extent 720×200 output.png
改一下图片的大小尺寸lsof –i
实时查看本机网络服务的活动状态。vim scp://username@host//path/to/somefile
vim一个远程文件
getopts
Note that getopts is not able to parse GNU-style long options (–myoption) or XF86-style long options (-myoption)!
- OPTIND Holds the index to the next argument to be processed. This is how getopts "remembers" its own status between invocations. Also usefull to shift the positional parameters after processing with getopts. OPTIND is initially set to 1, and needs to be re-set to 1 if you want to parse anything again with getopts
- OPTARG This variable is set to any argument for an option found by getopts. It also contains the option flag of an unknown option.
- OPTERR (Values 0 or 1) Indicates if Bash should display error messages generated by the getopts builtin. The value is initialized to 1 on every shell startup - so be sure to always set it to 0 if you don't want to see annoying messages!
#The base-syntax for getopts is: getopts OPTSTRING VARNAME [ARGS...] OPTSTRING tells getopts which options to expect and where to expect arguments (see below) VARNAME tells getopts which shell-variable to use for option reporting ARGS tells getopts to parse these optional words instead of the positional parameters
compute float number
result=$(echo 1 23 | awk '{printf("%.2f", ($1/$2)*100)}') =>result=4.35 a=12.34 b=23.45 echo "$a*$b"|bc a=12.34 b=23.45 echo "scale=1;$a/$b"|bc #保留小数点精度只对除法有效 .5 echo "scale=2;$a/$b"|bc .52
positional parameters
$FUNCNAME the function name (attention: inside a function, $0 is still the $0 of the shell, not the function name) $1 … $9 the argument list elements from 1 to 9 ${10} … ${N} the argument list elements beyond 9 (note the parameter expansion syntax!) $* all positional parameters except $0 $@ all positional parameters except $0 $# the number of arguments, not counting $0
loop through the positional parameters
numargs=$# for ((i=1 ; i <= numargs ; i++)) do echo "$1" shift done #or for arg do echo "$arg" done #while but it has the disadvantage to stop when $1 is empty (null-string) while [ "$1" ] do echo "$1" shift done # run as long as $1 is defined while [ "${1+defined}" ]; do echo "$1" shift done
mass usage
$* $1 $2 $3 … ${N} $@ $1 $2 $3 … ${N} "$*" "$1c$2c$3c…c${N}" # where 'c' is the first character of IFS. "$@" "$1" "$2" "$3" … "${N}"
Range Of Positional Parameters
${@:START:COUNT} ${*:START:COUNT} "${@:START:COUNT}" "${*:START:COUNT}"
多个文件总大小
cat filelist | xargs du -cb
diff ignore files only in one directory
diff -bBur old_dir/ new_dir/ | grep -v "^Only in" > my.diff
regular expression
匹配中文字符的正则表达式:[u4e00-u9fa5] 评注:匹配中文还真是个头疼的事,有了这个表达式就好办了 匹配双字节字符(包括汉字在内):[^x00-xff] 评注:可以用来计算字符串的长度(一个双字节字符长度计2,ASCII字符计1) 匹配空白行的正则表达式:ns*r 评注:可以用来删除空白行 匹配HTML标记的正则表达式:<(S*?)[^>]*>.*?</1>|<.*? /> 评注:网上流传的版本太糟糕,上面这个也仅仅能匹配部分,对于复杂的嵌套标记依旧无能为力 匹配首尾空白字符的正则表达式:^s*|s*$ 评注:可以用来删除行首行尾的空白字符(包括空格、制表符、换页符等等),非常有用的表达式 匹配Email地址的正则表达式:w+([-+.]w+)*@w+([-.]w+)*.w+([-.]w+)* 评注:表单验证时很实用 匹配网址URL的正则表达式:[a-zA-z]+://[^s]* 评注:网上流传的版本功能很有限,上面这个基本可以满足需求 匹配帐号是否合法(字母开头,允许5-16字节,允许字母数字下划线):^[a-zA-Z][a-zA-Z0-9_]{4,15}$ 评注:表单验证时很实用 匹配国内电话号码:d{3}-d{8}|d{4}-d{7} 评注:匹配形式如0511-4405222或021-87888822 匹配腾讯QQ号:[1-9][0-9]{4,} 评注:腾讯QQ号从10000开始 匹配中国邮政编码:[1-9]d{5}(?!d) 评注:中国邮政编码为6位数字 匹配身份证:d{15}|d{18} 评注:中国的身份证为15位或18位 匹配ip地址:d+.d+.d+.d+ 评注:提取ip地址时有用 匹配特定数字: ^[1-9]d*$ //匹配正整数 ^-[1-9]d*$ //匹配负整数 ^-?[1-9]d*$ //匹配整数 ^[1-9]d*|0$ //匹配非负整数(正整数+ 0) ^-[1-9]d*|0$ //匹配非正整数(负整数+ 0) ^[1-9]d*.d*|0.d*[1-9]d*$ //匹配正浮点数 ^-([1-9]d*.d*|0.d*[1-9]d*)$ //匹配负浮点数 ^-?([1-9]d*.d*|0.d*[1-9]d*|0?.0+|0)$ //匹配浮点数 ^[1-9]d*.d*|0.d*[1-9]d*|0?.0+|0$ //匹配非负浮点数(正浮点数+ 0) ^(-([1-9]d*.d*|0.d*[1-9]d*))|0?.0+|0$ //匹配非正浮点数(负浮点数+ 0) 评注:处理大量数据时有用,具体应用时注意修正 匹配特定字符串: ^[A-Za-z]+$ //匹配由26个英文字母组成的字符串 ^[A-Z]+$ //匹配由26个英文字母的大写组成的字符串 ^[a-z]+$ //匹配由26个英文字母的小写组成的字符串 ^[A-Za-z0-9]+$ //匹配由数字和26个英文字母组成的字符串 ^w+$ //匹配由数字、26个英文字母或者下划线组成的字符串
touch
touch all folders in a directory
find . -maxdepth 1 -mindepth 1 -type d -exec touch {} \+
touch all files in a directory
find * -exec touch {} \; # set time to 29th of Feb 2012. find * -exec touch -t 201202291000 {} \;
the path of directory in which a Bash script is located
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
If you want to also resolve any links to the script itself, you need a multi-line solution:
#!/bin/bash SOURCE="${BASH_SOURCE[0]}" while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink TARGET="$(readlink "$SOURCE")" if [[ $TARGET == /* ]]; then echo "SOURCE '$SOURCE' is an absolute symlink to '$TARGET'" SOURCE="$TARGET" else DIR="$( dirname "$SOURCE" )" echo "SOURCE '$SOURCE' is a relative symlink to '$TARGET' (relative to '$DIR')" SOURCE="$DIR/$TARGET" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located fi done echo "SOURCE is '$SOURCE'" RDIR="$( dirname "$SOURCE" )" DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" if [ "$DIR" != "$RDIR" ]; then echo "DIR '$RDIR' resolves to '$DIR'" fi echo "DIR is '$DIR'"
And it will print something like:
SOURCE './scriptdir.sh' is a relative symlink to 'sym2/scriptdir.sh' (relative to '.') SOURCE is './sym2/scriptdir.sh' DIR './sym2' resolves to '/home/ubuntu/dotfiles/fo fo/real/real1/real2' DIR is '/home/ubuntu/dotfiles/fo fo/real/real1/real2'
only one instance of a script is running at a time (mutual exclusion, locking
- uses a lockfile and echoes a PID into it
LOCKFILE=/tmp/lock.txt if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then echo "already running" exit fi # make sure the lockfile is removed when we exit and then claim it trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT echo $$ > ${LOCKFILE} # do stuff sleep 1000 rm -f ${LOCKFILE}
The trick here is the kill -0
which doesn't deliver any signal but
just checks if a process with the given PID exists. Also the call to
trap will ensure that the lockfile is removed even when your process
is killed (except kill -9
).
This example does not work, because there is a RaceCondition: a time window between checking and creating the file, during which other programs may act.3
- use
mkdir
We need an atomic check-and-create operation, and fortunately there is
one: mkdir
, the command to create a directory:
# POSIX (maybe Bourne?) lockdir=/tmp/myscript.lock if mkdir "$lockdir" then echo >&2 "successfully acquired lock" # Remove lockdir when the script finishes, or when it receives a signal trap 'rm -rf "$lockdir"' 0 # remove directory when script finishes # Optionally create temporary files in this directory, because # they will be removed automatically: tmpfile=$lockdir/filelist else echo >&2 "cannot acquire lock, giving up on $lockdir" exit 0 fi
Here, even when two processes call mkdir at the same time, only one process can succeed at most. This atomicity of check-and-create is ensured at the operating system kernel level.
This example is much better. There is still the problem that a stale lock could remain when the script is terminated with a signal not caught (or signal 9, SIGKILL), or could be created by a user (either accidentally or maliciously), but it's a good step towards reliable mutual exclusion. Charles Duffy has contributed an example that may remedy the "stale lock" problem.
- use
flock(1)
If you're using a GNU/Linux distribution, you can also get the benefit
of using flock(1)
. flock(1)
ties a FileDescriptor to a lock file.
There are multiple ways to use it; one possibility to solve the
multiple instance problem is:
exec 9>/path/to/lock/file if ! flock -n 9 ; then echo "another instance is running"; exit 1 fi # this now runs under the lock until 9 is closed (it will be closed automatically when the script ends)
flock can also be used to protect only a part of your script.
# mutex file # # Open a mutual exclusion lock on the file, unless another process already owns one. # # If the file is already locked by another process, the operation fails. # This function defines a lock on a file as having a file descriptor open to the file. # This function uses FD 9 to open a lock on the file. To release the lock, close FD 9: # exec 9>&- # mutex() { local file=$1 pid pids exec 9>>"$file" { pids=$(fuser -f "$file"); } 2>&- 9>&- for pid in $pids; do [[ $pid = $$ ]] && continue exec 9>&- return 1 # Locked by a pid. done } mutex /var/run/myscript.lock || { echo "Already running." >&2; exit 1; }